[2022-08-08 10:48:11.776249] WARN [SERVER] do_process (obmp_query.cpp:843) [11024][0][YB42C0A80B80-0005E3CF9BC835DD-0-0] [lt=4] [dc=0] query failed(ret=-5001, retry_ctrl_.need_retry()=0) [2022-08-08 10:48:11.776658] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10533) [2022-08-08 10:48:11.776871] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11457, time_dist: generate_cluster_version=11454) [2022-08-08 10:48:11.867015] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659926891867011, cur_ts_=1659926891838482, delta=28529) [2022-08-08 10:48:12.063893] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:12.064062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:12.065267] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:12.065314] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668391, retry interval=1000000, ret=-4179) [2022-08-08 10:48:12.282358] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45CC-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:12.306631] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:12.308149] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:12.308198] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668395, retry interval=1000000, ret=-4179) [2022-08-08 10:48:12.506957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28DD-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:12.507204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28DD-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:12.614387] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:12.651823] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:12.654976] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:12.655022] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:12.813782] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=21] [dc=0] [trans sampling] (TRACE=begin_ts=1659926892806511 2022-08-08 02:48:12.806511|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:12833238127466073404, inc:84065737, addr:{ip:"192.168.11.128", port:2882}, t:1659926892805000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB396-0-0, start_time:1659926892806386, snapshot:1659926892784770|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926892806654|[calc_checksum_by_commit] u=17 ret:0, checksum:2773129766, checksum_log_ts:9223372036854775807|[submit_log] u=27 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=7004 trans_version:1659926892805670, uref:1073741825|[exiting] u=26 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8312871, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=7231) [2022-08-08 10:48:12.827208] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:12.967372] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:12.994603] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=193] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14614, time_dist: do_cluster_heartbeat=9, generate_cluster_version=14601) [2022-08-08 10:48:13.064565] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.071935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:13.071992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:13.074975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.089063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:13.089263] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:13.090556] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:13.090627] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668390, retry interval=1000000, ret=-4179) [2022-08-08 10:48:13.129921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.247163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.259581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:13.269966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.281867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.292621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.301121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:13.328739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:13.332577] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:13.332615] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668394, retry interval=1000000, ret=-4179) [2022-08-08 10:48:13.333339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:13.403535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.435471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.439445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:13.448858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.465232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:13.467233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.468236] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.476725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:13.512375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.520439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.539746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.541257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:13.543658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.551521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:13.553581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.557597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.561499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.568004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.571677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.574412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.584830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.601797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.609140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.643083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.649795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.657741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.658322] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:13.665195] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:13.665232] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:13.666132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.671070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:13.694894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:13.696433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.705316] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.735933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.737799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:13.838911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:13.849344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.855590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:13.871562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.894535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:13.930641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.951316] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=209] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:13.997985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:14.018881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:14.070880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:14.162665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:14.162762] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:14.162813] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668389, retry interval=1000000, ret=-4179) [2022-08-08 10:48:14.163637] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926894163632, cur_ts_=1659926894135715, delta=27917) [2022-08-08 10:48:14.335305] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:14.360140] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:14.360306] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:14.361750] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:14.361802] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668393, retry interval=1000000, ret=-4179) [2022-08-08 10:48:14.514875] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.514892] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.514956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.515024] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.515145] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.515253] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.515432] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E3-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:14.668082] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:14.671775] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:14.671825] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:14.701111] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659926894701106, cur_ts_=1659926894674865, delta=26241) [2022-08-08 10:48:14.910433] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:48:15.186700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:15.186906] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:15.188567] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:15.188612] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668388, retry interval=1000000, ret=-4179) [2022-08-08 10:48:15.266407] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11473) [2022-08-08 10:48:15.266630] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15667, time_dist: generate_cluster_version=15665) [2022-08-08 10:48:15.401038] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14599, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14584) [2022-08-08 10:48:15.410573] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:15.410618] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668392, retry interval=1000000, ret=-4179) [2022-08-08 10:48:15.704430] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:15.707495] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:15.707528] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:16.198633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:16.198686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:16.244146] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926896244135, cur_ts_=1659926896212711, delta=31424) [2022-08-08 10:48:16.251406] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:16.252032] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=545] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:16.255210] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:16.255276] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668387, retry interval=1000000, ret=-4179) [2022-08-08 10:48:16.435231] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:16.435293] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668391, retry interval=1000000, ret=-4179) [2022-08-08 10:48:16.533943] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E9-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:16.533988] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E9-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:16.534139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:16.534273] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28E9-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:16.534392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:16.534547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:16.710425] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:16.713738] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:16.713809] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:17.283236] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:17.283298] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668386, retry interval=1000000, ret=-4179) [2022-08-08 10:48:17.437839] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4617-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:17.458533] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:17.459190] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=283] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:17.461466] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:17.461519] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668390, retry interval=1000000, ret=-4179) [2022-08-08 10:48:17.716654] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:17.719985] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:17.720037] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:18.305260] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] clock out of order(cur_ts=1659926898305252, cur_ts_=1659926898264794, delta=40458) [2022-08-08 10:48:18.307257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DDF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:18.307388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DDF-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:18.314766] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=52924) [2022-08-08 10:48:18.318292] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=57293, time_dist: generate_cluster_version=57046) [2022-08-08 10:48:18.324070] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20412, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20401) [2022-08-08 10:48:18.334926] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:18.334989] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668385, retry interval=1000000, ret=-4179) [2022-08-08 10:48:18.349078] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13306) [2022-08-08 10:48:18.436922] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10678) [2022-08-08 10:48:18.437154] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=376] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12287, time_dist: do_cluster_heartbeat=43, generate_cluster_version=12241) [2022-08-08 10:48:18.485595] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:18.485824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:18.487207] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:18.487269] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668389, retry interval=1000000, ret=-4179) [2022-08-08 10:48:18.492923] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11587) [2022-08-08 10:48:18.493547] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13469, time_dist: generate_cluster_version=13465) [2022-08-08 10:48:18.504582] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11564) [2022-08-08 10:48:18.547613] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28EF-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:18.547678] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28EF-0-0] [lt=46] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:18.547694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28EF-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:18.547702] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28EF-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:18.547804] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:18.664242] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12860) [2022-08-08 10:48:18.664655] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13948, time_dist: generate_cluster_version=13945) [2022-08-08 10:48:18.685027] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20740) [2022-08-08 10:48:18.694020] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43501, time_dist: generate_cluster_version=43491, cluster_service_self_check=5) [2022-08-08 10:48:18.724655] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:18.728313] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:18.728353] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:18.859505] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659926898858561 2022-08-08 02:48:18.858561|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:1248289653762517495, inc:84065971, addr:{ip:"192.168.11.128", port:2882}, t:1659926898858000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A55ED-0-0, start_time:1659926898858423, snapshot:1659926898823829|[end_task] u=141 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926898858719|[calc_checksum_by_commit] u=20 ret:0, checksum:3372293884, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=640 trans_version:1659926898858740, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8298315, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=894) [2022-08-08 10:48:19.267119] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.295669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.337653] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45D7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:19.337685] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45D7-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:19.357287] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:19.357892] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:19.357925] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668384, retry interval=1000000, ret=-4179) [2022-08-08 10:48:19.407706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.513381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:19.513418] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:19.514806] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:19.514852] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668388, retry interval=1000000, ret=-4179) [2022-08-08 10:48:19.516379] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10385) [2022-08-08 10:48:19.597136] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:48:19.597201] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:48:19.636988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:19.659101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.659162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.659231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:19.679970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:19.709938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:19.715927] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14545472}) [2022-08-08 10:48:19.730059] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:19.731221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.733164] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:19.733183] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:19.758346] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:19.777107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:19.777318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:19.777332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:19.900539] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:19.910358] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926898170564, end_time=1659926899910346, elapsed_time=1739782, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:48:19.938009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:20.104560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:20.146179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:20.218939] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659926900218929, cur_ts_=1659926900186928, delta=32001) [2022-08-08 10:48:20.229073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:20.360445] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C461A-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.360530] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C461A-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.360686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C461A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.360859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C461A-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.361008] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C461A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.361193] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C461A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.384049] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.384079] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:20.385399] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:20.385454] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668383, retry interval=1000000, ret=-4179) [2022-08-08 10:48:20.505984] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=692] [dc=0] clock out of order(cur_ts=1659926900505976, cur_ts_=1659926900469047, delta=36929) [2022-08-08 10:48:20.571734] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28F5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.571808] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28F5-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.572001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28F5-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.572112] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE28F5-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.572257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.572444] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.572477] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.572587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:20.593385] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:20.593424] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668387, retry interval=1000000, ret=-4179) [2022-08-08 10:48:20.735426] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:20.740147] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:20.740185] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:20.852378] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17451) [2022-08-08 10:48:20.853590] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18091, time_dist: do_cluster_heartbeat=9, generate_cluster_version=18078) [2022-08-08 10:48:20.856194] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22874, time_dist: do_cluster_heartbeat=8, generate_cluster_version=22863) [2022-08-08 10:48:20.881511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:21.415953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:21.416091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:21.417476] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:21.417537] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668382, retry interval=1000000, ret=-4179) [2022-08-08 10:48:21.606982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:21.607010] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:21.622612] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:21.624091] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:21.624150] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668386, retry interval=1000000, ret=-4179) [2022-08-08 10:48:21.742308] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:21.745417] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:21.745453] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:21.897463] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] [trans sampling] (TRACE=begin_ts=1659926901896585 2022-08-08 02:48:21.896585|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:5493931572121295745, inc:84066089, addr:{ip:"192.168.11.128", port:2882}, t:1659926901895000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB3C4-0-0, start_time:1659926901896422, snapshot:1659926901849350|[end_task] u=134 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926901896735|[calc_checksum_by_commit] u=18 ret:0, checksum:661399394, checksum_log_ts:9223372036854775807|[submit_log] u=28 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=608 trans_version:1659926901895221, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8312917, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=839) [2022-08-08 10:48:22.419554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DE5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:22.419879] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DE5-0-0] [lt=48] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:22.420153] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DE5-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:22.440315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:22.444524] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=205] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:22.444674] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] task process failed, start retry(max retry time=9223372036854668381, retry interval=1000000, ret=-4179) [2022-08-08 10:48:22.654774] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:22.654839] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668385, retry interval=1000000, ret=-4179) [2022-08-08 10:48:22.747741] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:22.751104] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:22.751356] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:22.888123] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:22.990886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:23.084362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.099869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.146565] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.265142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.282079] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:23.289652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.303241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.311660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.321111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:23.349164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:23.418882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.454746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.463608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:23.470213] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:23.474429] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:23.474497] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668380, retry interval=1000000, ret=-4179) [2022-08-08 10:48:23.475269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.484800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.485732] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:23.490591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.495401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:23.536842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.540601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.557821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:23.561182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.563850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.574033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:23.574147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.582484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.583011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.584999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.590895] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.592421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.597556] WARN [SQL] execute_get_plan (ob_sql.cpp:3081) [11024][0][YB42C0A80B80-0005E3CF9BC835DE-0-0] [lt=46] [dc=0] fail to get plan retry(ret=-5138) [2022-08-08 10:48:23.606036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.626377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.634094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.667361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.667479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.677933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.681904] WARN [SHARE] alloc_handle (ob_autoincrement_service.cpp:100) [11024][0][YB42C0A80B80-0005E3CF9BC835DE-0-0] [lt=35] [dc=0] fail to alloc handle; cache is not enough(*this={table_id:{value:0, first:0, second:0}, next_value:0, local_sync:0, last_refresh_ts:1659926903681885, curr_node:{cache_start:0, cache_end:0}, prefetch_node:{cache_start:0, cache_end:0}}, ret=-4019) [2022-08-08 10:48:23.699017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.704144] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:23.714793] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15190, time_dist: generate_cluster_version=15188) [2022-08-08 10:48:23.719644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:23.721578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.721721] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.724890] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10490) [2022-08-08 10:48:23.725139] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20763, time_dist: generate_cluster_version=20761) [2022-08-08 10:48:23.755480] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:23.755544] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668384, retry interval=1000000, ret=-4179) [2022-08-08 10:48:23.756619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:23.756741] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:23.760569] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:23.760608] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:23.760691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=140] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.854394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:23.864784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.871637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:23.891647] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.916881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:23.947572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:23.966899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:24.015053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:24.044175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:24.094455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:24.187618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:24.226055] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659926904226047, cur_ts_=1659926904196575, delta=29472) [2022-08-08 10:48:24.355067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:24.476271] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.476434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.476690] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.476873] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.477099] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.477114] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.477249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.477381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.477492] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4620-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.477863] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.496918] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:24.496955] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668379, retry interval=1000000, ret=-4179) [2022-08-08 10:48:24.763985] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:24.767313] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:24.767336] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:24.776315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.776347] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:24.778270] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:24.778423] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668383, retry interval=1000000, ret=-4179) [2022-08-08 10:48:24.908858] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] [trans sampling] (TRACE=begin_ts=1659926904907999 2022-08-08 02:48:24.907999|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:8761494371623699662, inc:84066210, addr:{ip:"192.168.11.128", port:2882}, t:1659926904907000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A560B-0-0, start_time:1659926904907882, snapshot:1659926904848387|[end_task] u=122 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926904908137|[calc_checksum_by_commit] u=20 ret:0, checksum:1128171628, checksum_log_ts:9223372036854775807|[submit_log] u=50 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=562 trans_version:1659926904908159, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8298345, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=818) [2022-08-08 10:48:25.519779] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:25.519928] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668378, retry interval=1000000, ret=-4179) [2022-08-08 10:48:25.772989] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:25.776382] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:25.776421] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:25.784288] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:25.784313] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:25.812579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:25.812627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:25.814647] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:25.814719] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668382, retry interval=1000000, ret=-4179) [2022-08-08 10:48:26.540892] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:26.541064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:26.542470] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:26.542510] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668377, retry interval=1000000, ret=-4179) [2022-08-08 10:48:26.779285] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:26.796864] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:26.796903] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:26.833002] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:26.833038] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:26.835731] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:26.835797] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668381, retry interval=1000000, ret=-4179) [2022-08-08 10:48:27.562895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:27.564731] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:27.564771] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668376, retry interval=1000000, ret=-4179) [2022-08-08 10:48:27.799210] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:27.802842] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:27.802887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:27.862852] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:27.862891] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668380, retry interval=1000000, ret=-4179) [2022-08-08 10:48:27.943759] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] clock out of order(cur_ts=1659926907943751, cur_ts_=1659926907903715, delta=40036) [2022-08-08 10:48:27.962051] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15788) [2022-08-08 10:48:27.966699] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26153, time_dist: do_cluster_heartbeat=11, generate_cluster_version=26138) [2022-08-08 10:48:27.967745] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21622, time_dist: do_cluster_heartbeat=8, generate_cluster_version=21610) [2022-08-08 10:48:28.588054] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:28.588301] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=148] [dc=0] task process failed, start retry(max retry time=9223372036854668375, retry interval=1000000, ret=-4179) [2022-08-08 10:48:28.654723] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE290D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:28.654753] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE290D-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:28.654768] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE290D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:28.655096] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE290D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:28.805492] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:28.809073] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:28.809097] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:28.976663] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926908976653, cur_ts_=1659926908938587, delta=38066) [2022-08-08 10:48:28.982273] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=48601) [2022-08-08 10:48:28.997965] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:28.998017] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668379, retry interval=1000000, ret=-4179) [2022-08-08 10:48:28.999155] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16842) [2022-08-08 10:48:28.999593] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=66809, time_dist: do_cluster_heartbeat=4, generate_cluster_version=66802) [2022-08-08 10:48:29.000196] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28486, time_dist: generate_cluster_version=28484) [2022-08-08 10:48:29.287731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.315513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.423802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.606753] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:48:29.606777] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:48:29.609093] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:29.609161] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668374, retry interval=1000000, ret=-4179) [2022-08-08 10:48:29.653639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:29.656302] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659926909656297, cur_ts_=1659926909630475, delta=25822) [2022-08-08 10:48:29.679672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.679777] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:29.683092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.701864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:29.729632] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:29.750468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.778324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:29.793004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:29.795040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:29.797276] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14549888}) [2022-08-08 10:48:29.802505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:29.821771] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:29.856042] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926909856033, cur_ts_=1659926909821255, delta=34778) [2022-08-08 10:48:29.857244] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:29.857281] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:29.917630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:29.919623] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10710) [2022-08-08 10:48:29.926082] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18365, time_dist: do_cluster_heartbeat=8, generate_cluster_version=18351) [2022-08-08 10:48:29.928044] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=208] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20312, time_dist: do_cluster_heartbeat=5, generate_cluster_version=20302) [2022-08-08 10:48:29.954214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:30.084467] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926908171562, end_time=1659926910084455, elapsed_time=1912893, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:48:30.093561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:30.093763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:30.094781] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:30.094841] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668378, retry interval=1000000, ret=-4179) [2022-08-08 10:48:30.125010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:30.171942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:30.245056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:30.633057] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:30.633138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:30.634686] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:30.634724] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668373, retry interval=1000000, ret=-4179) [2022-08-08 10:48:30.887236] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:30.890290] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:30.890328] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:31.131415] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:31.131450] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668377, retry interval=1000000, ret=-4179) [2022-08-08 10:48:31.655241] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:31.656743] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:31.656797] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668372, retry interval=1000000, ret=-4179) [2022-08-08 10:48:31.895543] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:31.898843] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:31.898888] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:32.134362] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF4-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.134399] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF4-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.134411] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.134487] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF4-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.134771] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.134941] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF4-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.135336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.153637] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:32.153689] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668376, retry interval=1000000, ret=-4179) [2022-08-08 10:48:32.615360] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:32.682314] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:32.684024] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=174] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:32.685065] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=845] [dc=0] task process failed, start retry(max retry time=9223372036854668371, retry interval=1000000, ret=-4179) [2022-08-08 10:48:32.914020] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:32.925267] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:32.925296] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:32.953789] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:33.012562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:33.100900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.116262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.169403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.178847] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:33.178977] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:33.180606] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:33.180686] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668375, retry interval=1000000, ret=-4179) [2022-08-08 10:48:33.289873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.306328] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:33.306401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.327739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.328523] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.338108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:33.369352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:33.444857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.473432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.487935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:33.492670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.502604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.508533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:33.515304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.515377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:33.560664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.563110] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.579045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.582470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.582546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:33.598262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.599075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.599103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:33.602302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.603943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.609256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.614577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.631163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.643854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.649925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.684653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.689828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.698228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.717375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.718883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:33.719007] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:33.719964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:33.720460] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:33.720522] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668370, retry interval=1000000, ret=-4179) [2022-08-08 10:48:33.736314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:33.739204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.741714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.779070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:33.779923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.871192] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:33.883761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.892080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:33.915949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.927672] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:33.930881] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:33.930917] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:33.933086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:33.972204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=209] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:33.989497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:34.030910] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:34.069010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:34.112711] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:34.183987] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C462F-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.184028] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C462F-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.204144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.205721] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:34.205783] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668374, retry interval=1000000, ret=-4179) [2022-08-08 10:48:34.206500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:34.376757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:34.726272] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.726455] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=159] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.726563] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=79] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.726616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=42] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.726802] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=101] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.727056] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.727232] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.727370] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5DF7-0-0] [lt=79] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.744989] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:34.746397] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:34.746449] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668369, retry interval=1000000, ret=-4179) [2022-08-08 10:48:34.936728] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:34.938759] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:48:34.940673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:34.940707] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:35.209027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4771-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:35.209330] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4771-0-0] [lt=231] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:35.210212] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:35.228374] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:35.228484] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:35.229902] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:35.230031] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668373, retry interval=1000000, ret=-4179) [2022-08-08 10:48:35.770558] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:35.770599] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668368, retry interval=1000000, ret=-4179) [2022-08-08 10:48:35.944952] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:35.949033] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:35.949058] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:36.274106] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:36.274150] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668372, retry interval=1000000, ret=-4179) [2022-08-08 10:48:36.339215] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926916339206, cur_ts_=1659926916313933, delta=25273) [2022-08-08 10:48:36.773001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4632-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.773211] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4632-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.773309] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4632-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.775725] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.775749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.793457] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.793703] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:36.795003] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:36.795056] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668367, retry interval=1000000, ret=-4179) [2022-08-08 10:48:36.951799] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:36.954815] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:36.954851] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:37.298853] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:37.299699] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:37.301995] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:37.302040] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668371, retry interval=1000000, ret=-4179) [2022-08-08 10:48:37.489269] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926917489264, cur_ts_=1659926917466478, delta=22786) [2022-08-08 10:48:37.814686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:37.819106] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:37.819355] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668366, retry interval=1000000, ret=-4179) [2022-08-08 10:48:37.957131] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:37.960447] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:37.960481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:38.002061] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926918002054, cur_ts_=1659926917972743, delta=29311) [2022-08-08 10:48:38.329789] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:38.329839] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854668370, retry interval=1000000, ret=-4179) [2022-08-08 10:48:38.364832] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13510) [2022-08-08 10:48:38.371377] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21025, time_dist: generate_cluster_version=21020) [2022-08-08 10:48:38.849799] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926918849790, cur_ts_=1659926918820433, delta=29357) [2022-08-08 10:48:38.874735] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:38.874911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:38.875580] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:38.875638] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668365, retry interval=1000000, ret=-4179) [2022-08-08 10:48:38.964294] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:38.967324] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:38.967349] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:39.304822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.335936] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.355500] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:39.355670] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:39.356669] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:39.356745] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668369, retry interval=1000000, ret=-4179) [2022-08-08 10:48:39.446926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.533491] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659926919533485, cur_ts_=1659926919512428, delta=21057) [2022-08-08 10:48:39.598461] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:48:39.598492] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:48:39.659205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:39.671432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:39.702523] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.702897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:39.704518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.724564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:39.752494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:39.767554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.798164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:39.811182] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14554208}) [2022-08-08 10:48:39.815620] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:39.820180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:39.820226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:39.901858] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:39.901922] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668364, retry interval=1000000, ret=-4179) [2022-08-08 10:48:39.939914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:39.969210] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:39.981010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:39.985723] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:39.985749] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:40.016485] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926918177215, end_time=1659926920016471, elapsed_time=1839256, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:48:40.155893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:40.197333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:40.262340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:40.386150] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:40.386201] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668368, retry interval=1000000, ret=-4179) [2022-08-08 10:48:40.933788] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:40.933843] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668363, retry interval=1000000, ret=-4179) [2022-08-08 10:48:40.991370] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:40.994574] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:40.994615] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:41.389386] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.389554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.389756] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.389876] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.390158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.390240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.390440] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.391243] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C45F8-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.391788] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.415889] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659926921415882, cur_ts_=1659926921391615, delta=24267) [2022-08-08 10:48:41.440822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.441081] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.445128] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:41.445199] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668367, retry interval=1000000, ret=-4179) [2022-08-08 10:48:41.588056] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659926921587505, cur_ts_=1659926921564339, delta=23166) [2022-08-08 10:48:41.698800] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659926921698795, cur_ts_=1659926921676071, delta=22724) [2022-08-08 10:48:41.803439] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926921803435, cur_ts_=1659926921771191, delta=32244) [2022-08-08 10:48:41.947366] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.947893] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.975200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.975232] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:41.976736] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:41.976797] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668362, retry interval=1000000, ret=-4179) [2022-08-08 10:48:41.996852] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:42.000552] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:42.000596] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:42.048656] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18879) [2022-08-08 10:48:42.049015] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20633, time_dist: generate_cluster_version=20630) [2022-08-08 10:48:42.465832] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:42.465994] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:42.467418] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:42.467467] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668366, retry interval=1000000, ret=-4179) [2022-08-08 10:48:42.764061] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2937-0-0] [lt=46] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.764199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2937-0-0] [lt=110] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.764421] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2937-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.764593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2937-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.764790] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.764951] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.765073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:42.993311] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:42.993356] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:43.006514] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:43.016414] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:43.020829] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:43.020873] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:43.025073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:43.025096] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:43.027227] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:43.027268] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668361, retry interval=1000000, ret=-4179) [2022-08-08 10:48:43.035262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:43.145663] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659926923145656, cur_ts_=1659926923116800, delta=28856) [2022-08-08 10:48:43.145790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.153820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.188808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.234592] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20073, time_dist: do_cluster_heartbeat=10, generate_cluster_version=20058) [2022-08-08 10:48:43.235765] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20447, time_dist: do_cluster_heartbeat=5, generate_cluster_version=20438) [2022-08-08 10:48:43.264482] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659926923264473, cur_ts_=1659926923236857, delta=27616) [2022-08-08 10:48:43.308693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.328738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:43.331670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.348660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.349262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.359167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:43.394461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:43.469702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.470017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C477D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:43.470043] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C477D-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:43.473382] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:43.490421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.491630] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:43.491683] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668365, retry interval=1000000, ret=-4179) [2022-08-08 10:48:43.517429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.517522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:43.521842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.532069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:43.532258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.538126] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:43.584180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.584382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.594812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.600468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.602043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:43.614220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.615779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.634202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.634300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.634319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.634402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.634524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:43.655702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.665632] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=860] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.667985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=493] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.705582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.710275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.719476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.741386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:43.746839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.775639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:43.775668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.775828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.776340] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926923776334, cur_ts_=1659926923743749, delta=32585) [2022-08-08 10:48:43.796966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.798402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:43.891780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:43.905513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.915887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:43.935735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:43.956172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:43.987414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:44.008162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:44.024231] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:44.041626] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:44.041648] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:44.053640] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:44.069781] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:44.071369] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:44.071431] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668360, retry interval=1000000, ret=-4179) [2022-08-08 10:48:44.084443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:44.135194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:44.236020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:44.396598] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:44.493965] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E06-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:44.494123] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E06-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:44.514969] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:44.515190] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:44.516540] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:44.516591] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668364, retry interval=1000000, ret=-4179) [2022-08-08 10:48:44.718474] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:48:45.044394] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:45.047850] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:45.047886] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:45.105049] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:45.106598] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:45.106763] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] task process failed, start retry(max retry time=9223372036854668359, retry interval=1000000, ret=-4179) [2022-08-08 10:48:45.249987] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10077, time_dist: generate_cluster_version=10073) [2022-08-08 10:48:45.542072] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:45.543864] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:45.543913] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668363, retry interval=1000000, ret=-4179) [2022-08-08 10:48:45.661866] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2018] [dc=0] clock out of order(cur_ts=1659926925661859, cur_ts_=1659926925627939, delta=33920) [2022-08-08 10:48:45.664941] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35484, time_dist: generate_cluster_version=35481) [2022-08-08 10:48:45.797270] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2940-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:46.034632] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926926034628, cur_ts_=1659926926000101, delta=34527) [2022-08-08 10:48:46.054814] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:46.058402] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:46.058439] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:46.139201] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659926926139197, cur_ts_=1659926926107501, delta=31696) [2022-08-08 10:48:46.147218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:46.147286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:46.165408] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:46.165669] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:46.166694] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:46.166734] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668358, retry interval=1000000, ret=-4179) [2022-08-08 10:48:46.168080] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659926926167150 2022-08-08 02:48:46.167150|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:14084331175784766417, inc:84067011, addr:{ip:"192.168.11.128", port:2882}, t:1659926926166000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=5 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F78E9-0-0, start_time:1659926926167043, snapshot:1659926926141243|[end_task] u=165 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926926167331|[calc_checksum_by_commit] u=16 ret:0, checksum:1224488723, checksum_log_ts:9223372036854775807|[submit_log] u=62 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=599 trans_version:1659926926166782, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:986802, logging:0, pending:0, uref:1|[end_trans_cb] u=15 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=894) [2022-08-08 10:48:46.570390] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926926570382, cur_ts_=1659926926543170, delta=27212) [2022-08-08 10:48:46.592052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:46.593875] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:46.593928] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668362, retry interval=1000000, ret=-4179) [2022-08-08 10:48:47.061047] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:47.064720] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:47.064840] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:47.194266] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:47.194480] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] task process failed, start retry(max retry time=9223372036854668357, retry interval=1000000, ret=-4179) [2022-08-08 10:48:47.483046] WARN [SQL] execute_get_plan (ob_sql.cpp:3081) [11024][0][YB42C0A80B80-0005E3CF9BC835E1-0-0] [lt=16] [dc=0] fail to get plan retry(ret=-5138) [2022-08-08 10:48:47.497585] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835E1-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:48:47.497845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835E1-0-0] [lt=184] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:48:47.497883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835E1-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:48:47.498227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835E1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:48:47.526106] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835E1-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:48:47.622471] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:47.622524] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668361, retry interval=1000000, ret=-4179) [2022-08-08 10:48:47.875172] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18529, time_dist: generate_cluster_version=18526) [2022-08-08 10:48:48.070447] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:48.073716] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:48.073759] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:48.199341] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:48.199370] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:48.227677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:48.227877] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:48.230142] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:48.230196] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668356, retry interval=1000000, ret=-4179) [2022-08-08 10:48:48.275813] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926928275804, cur_ts_=1659926928251429, delta=24375) [2022-08-08 10:48:48.454796] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14133) [2022-08-08 10:48:48.455142] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15269, time_dist: do_cluster_heartbeat=10, generate_cluster_version=15256) [2022-08-08 10:48:48.633436] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16466) [2022-08-08 10:48:48.634513] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18331, time_dist: generate_cluster_version=18328) [2022-08-08 10:48:48.650223] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:48.650259] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668360, retry interval=1000000, ret=-4179) [2022-08-08 10:48:48.650416] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16080) [2022-08-08 10:48:48.651577] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30741, time_dist: generate_cluster_version=30739) [2022-08-08 10:48:48.861288] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13503, time_dist: generate_cluster_version=13499) [2022-08-08 10:48:49.081285] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:49.090231] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:49.090269] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:49.258882] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:49.265083] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:49.265132] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668355, retry interval=1000000, ret=-4179) [2022-08-08 10:48:49.327484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:49.353676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:49.464694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:49.596639] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:48:49.598199] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=1552] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:48:49.659960] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:49.659988] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:49.677311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:49.680921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:49.682112] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:49.682151] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668359, retry interval=1000000, ret=-4179) [2022-08-08 10:48:49.687740] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:49.718245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=549] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:49.721334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:49.728668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:49.742071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:49.775321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:49.792163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:49.817488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:49.835652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:49.842315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:49.848971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:49.887364] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926929887359, cur_ts_=1659926929848624, delta=38735) [2022-08-08 10:48:49.931095] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=87459) [2022-08-08 10:48:49.932008] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=47644, time_dist: do_cluster_heartbeat=6, generate_cluster_version=47636) [2022-08-08 10:48:49.932082] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14558624}) [2022-08-08 10:48:49.945574] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=103169, time_dist: do_cluster_heartbeat=10, generate_cluster_version=103156) [2022-08-08 10:48:49.958758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:50.002755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:50.091907] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:50.109081] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17931, time_dist: generate_cluster_version=17927) [2022-08-08 10:48:50.109218] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:50.109253] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:50.134592] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926928183012, end_time=1659926930134576, elapsed_time=1951564, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:48:50.176974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:50.219239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:50.284576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:50.291139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:50.291322] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:50.293069] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:50.293123] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668354, retry interval=1000000, ret=-4179) [2022-08-08 10:48:50.704967] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:50.705185] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:50.706794] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:50.706874] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] task process failed, start retry(max retry time=9223372036854668358, retry interval=1000000, ret=-4179) [2022-08-08 10:48:51.112998] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:51.117413] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:51.117452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:51.315708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:51.315845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:51.319499] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:51.319552] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668353, retry interval=1000000, ret=-4179) [2022-08-08 10:48:51.408317] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926931408311, cur_ts_=1659926931385651, delta=22660) [2022-08-08 10:48:51.712356] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:51.734000] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:51.734200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:51.735914] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:51.735949] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668357, retry interval=1000000, ret=-4179) [2022-08-08 10:48:51.941321] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659926931941313, cur_ts_=1659926931914917, delta=26396) [2022-08-08 10:48:52.119300] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:52.125355] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:52.125394] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:52.322619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.325410] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.325571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.325683] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.325829] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.325999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.326145] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.326352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.326445] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4608-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.356514] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:52.356551] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668352, retry interval=1000000, ret=-4179) [2022-08-08 10:48:52.615612] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:52.764481] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:52.765743] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:52.765836] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668356, retry interval=1000000, ret=-4179) [2022-08-08 10:48:53.004198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.004217] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.004222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.004227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.004505] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.004526] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.004731] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2955-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.006348] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:48:53.056088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:53.082329] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:53.107457] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24089) [2022-08-08 10:48:53.130669] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:53.133858] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:53.133897] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:53.163074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.173142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.205956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.328832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.350633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.351599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:53.371528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.371920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.382329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:53.385896] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:53.387114] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:53.387175] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668351, retry interval=1000000, ret=-4179) [2022-08-08 10:48:53.417789] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:53.491965] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.510973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.532683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:53.542096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.543076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.550692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.553552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:53.556414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:53.602236] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.605766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.610206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.640494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.640675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=205] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.641038] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659926933641033, cur_ts_=1659926933612827, delta=28206) [2022-08-08 10:48:53.641138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:53.650917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.650928] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.651072] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.651095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.651129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.651822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:48:53.672095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.682257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.683276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.724761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.735182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.744202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.764373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.766487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:53.788776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:53.788921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:53.790385] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:53.790498] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] task process failed, start retry(max retry time=9223372036854668355, retry interval=1000000, ret=-4179) [2022-08-08 10:48:53.797086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.797583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.797628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:53.819620] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:53.819739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.911761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:53.922158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.939029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:53.958969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:53.973195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:54.009263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:54.025119] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:54.078885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:54.104381] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:48:54.136349] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:54.139605] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:54.139652] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:54.151920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:54.253330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:54.392139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:54.392214] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:54.409753] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:54.410437] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:54.414536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:54.415874] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:54.415944] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668350, retry interval=1000000, ret=-4179) [2022-08-08 10:48:54.724350] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16816, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16802) [2022-08-08 10:48:54.724861] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16475, time_dist: do_cluster_heartbeat=6, generate_cluster_version=16466) [2022-08-08 10:48:54.821381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:54.821421] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:54.823555] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:54.823621] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668354, retry interval=1000000, ret=-4179) [2022-08-08 10:48:54.966580] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:48:55.142338] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:55.146841] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:55.146878] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:55.422947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4650-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:55.423053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4650-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:55.428976] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:55.429021] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:55.448888] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:55.448991] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] task process failed, start retry(max retry time=9223372036854668349, retry interval=1000000, ret=-4179) [2022-08-08 10:48:55.846130] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:55.849537] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:55.849577] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668353, retry interval=1000000, ret=-4179) [2022-08-08 10:48:56.149601] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:56.152754] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:56.152799] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:56.469867] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:56.473809] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:56.473855] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668348, retry interval=1000000, ret=-4179) [2022-08-08 10:48:56.874448] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:56.876348] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:56.876553] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668352, retry interval=1000000, ret=-4179) [2022-08-08 10:48:57.155436] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:57.159192] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:57.159250] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:57.305647] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26989) [2022-08-08 10:48:57.305971] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27937, time_dist: generate_cluster_version=27936) [2022-08-08 10:48:57.313290] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35367, time_dist: do_cluster_heartbeat=13, generate_cluster_version=35351) [2022-08-08 10:48:57.388776] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19117) [2022-08-08 10:48:57.389053] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20227, time_dist: generate_cluster_version=20220, cluster_service_self_check=4) [2022-08-08 10:48:57.492162] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16491) [2022-08-08 10:48:57.492507] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18568, time_dist: do_cluster_heartbeat=11, generate_cluster_version=18555) [2022-08-08 10:48:57.514179] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:57.515765] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:57.515830] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668347, retry interval=1000000, ret=-4179) [2022-08-08 10:48:57.884468] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:57.902884] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:57.904077] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:57.904114] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668351, retry interval=1000000, ret=-4179) [2022-08-08 10:48:57.926104] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12185, time_dist: generate_cluster_version=12182) [2022-08-08 10:48:58.074313] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=373] [dc=0] clock out of order(cur_ts=1659926938074307, cur_ts_=1659926938051300, delta=23007) [2022-08-08 10:48:58.163809] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:58.167105] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:58.167161] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:58.303840] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10468) [2022-08-08 10:48:58.304043] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11569, time_dist: generate_cluster_version=11568) [2022-08-08 10:48:58.324976] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21099) [2022-08-08 10:48:58.325165] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21989, time_dist: generate_cluster_version=21987) [2022-08-08 10:48:58.478803] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12788) [2022-08-08 10:48:58.479265] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14554, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14541) [2022-08-08 10:48:58.502286] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12264) [2022-08-08 10:48:58.503658] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14788, time_dist: generate_cluster_version=14784) [2022-08-08 10:48:58.526419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:58.526451] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:58.560173] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:58.560243] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668346, retry interval=1000000, ret=-4179) [2022-08-08 10:48:58.588499] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27555) [2022-08-08 10:48:58.727788] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12601) [2022-08-08 10:48:58.728224] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31864, time_dist: generate_cluster_version=31862) [2022-08-08 10:48:58.741874] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14004) [2022-08-08 10:48:58.742112] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1634] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15954, time_dist: generate_cluster_version=15952) [2022-08-08 10:48:58.945530] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31085, time_dist: do_cluster_heartbeat=12, generate_cluster_version=31068) [2022-08-08 10:48:58.953762] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:58.953945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:48:58.954771] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:58.959694] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854668350, retry interval=1000000, ret=-4179) [2022-08-08 10:48:59.065786] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2466] [dc=0] clock out of order(cur_ts=1659926939065781, cur_ts_=1659926939032064, delta=33717) [2022-08-08 10:48:59.169919] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:48:59.179465] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:48:59.180481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=999] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:48:59.189445] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12378) [2022-08-08 10:48:59.202421] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28091, time_dist: generate_cluster_version=28088) [2022-08-08 10:48:59.352865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:59.377324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:59.469078] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926939469069, cur_ts_=1659926939434527, delta=34542) [2022-08-08 10:48:59.487352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:59.550486] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10797, time_dist: generate_cluster_version=10793) [2022-08-08 10:48:59.594008] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:48:59.594395] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=335] [dc=0] task process failed, start retry(max retry time=9223372036854668345, retry interval=1000000, ret=-4179) [2022-08-08 10:48:59.597452] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=394] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:48:59.597471] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:48:59.693441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:48:59.703737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:59.742084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:59.742133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:59.750052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:59.762941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:59.793171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:48:59.815188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:48:59.833572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:59.852067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:59.862237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:48:59.873511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:48:59.916089] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659926939916081, cur_ts_=1659926939885004, delta=31077) [2022-08-08 10:48:59.978865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:00.000887] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18902) [2022-08-08 10:49:00.009863] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29144, time_dist: do_cluster_heartbeat=13, generate_cluster_version=29035) [2022-08-08 10:49:00.027024] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:00.050272] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=184] [dc=0] clock out of order(cur_ts=1659926940050262, cur_ts_=1659926940024945, delta=25317) [2022-08-08 10:49:00.056283] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=338] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:00.056380] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668349, retry interval=1000000, ret=-4179) [2022-08-08 10:49:00.071779] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14563168}) [2022-08-08 10:49:00.123783] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE296A-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:00.123848] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE296A-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:00.124136] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:00.198402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:00.202491] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=249] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:00.205979] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:00.206005] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:00.237964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:00.305122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:00.339815] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926938187584, end_time=1659926940339804, elapsed_time=2152220, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:00.505173] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926940505163, cur_ts_=1659926940480165, delta=24998) [2022-08-08 10:49:00.622621] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:00.622666] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:00.624766] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:00.624819] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668344, retry interval=1000000, ret=-4179) [2022-08-08 10:49:01.059187] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E1E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:01.059226] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E1E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:01.061426] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:01.084589] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:01.084653] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668348, retry interval=1000000, ret=-4179) [2022-08-08 10:49:01.219162] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:01.225817] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:01.225866] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:01.654696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:01.656171] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:01.656233] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668343, retry interval=1000000, ret=-4179) [2022-08-08 10:49:02.009652] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11937) [2022-08-08 10:49:02.009930] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12939, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12928) [2022-08-08 10:49:02.111693] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:02.111754] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668347, retry interval=1000000, ret=-4179) [2022-08-08 10:49:02.145071] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:02.145336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:02.145556] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:02.145784] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:02.228623] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:02.232600] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:02.232659] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:02.572351] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] clock out of order(cur_ts=1659926942572346, cur_ts_=1659926942547084, delta=25262) [2022-08-08 10:49:02.645498] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12164) [2022-08-08 10:49:02.646685] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14262, time_dist: generate_cluster_version=14256) [2022-08-08 10:49:02.654824] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20444, time_dist: generate_cluster_version=20441) [2022-08-08 10:49:02.685098] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:02.685155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:02.719286] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:02.719325] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668342, retry interval=1000000, ret=-4179) [2022-08-08 10:49:02.793550] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926942793542, cur_ts_=1659926942761758, delta=31784) [2022-08-08 10:49:02.796840] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34857, time_dist: generate_cluster_version=34853) [2022-08-08 10:49:02.797051] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=37084, time_dist: do_cluster_heartbeat=11, generate_cluster_version=37070) [2022-08-08 10:49:03.078894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:03.163677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2973-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.163715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2973-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.164950] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2973-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.164976] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2973-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.165077] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.165094] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.165108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.165842] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:03.172119] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=147] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:03.172224] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] task process failed, start retry(max retry time=9223372036854668346, retry interval=1000000, ret=-4179) [2022-08-08 10:49:03.182700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.185641] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:03.189302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.231223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.237720] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:03.243763] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:03.243813] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:03.354499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.372206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:03.372429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.401171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.401171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.401600] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1156] [dc=0] clock out of order(cur_ts=1659926943401597, cur_ts_=1659926943380258, delta=21339) [2022-08-08 10:49:03.404053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:03.436881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:03.513747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.527628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.550046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:03.565850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.566138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.569550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.577297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:03.591136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:03.622403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.628715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.631399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.658782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=870] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.660182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=278] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:03.662739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=719] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.670278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.670783] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.677359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.677848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:03.678021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.678023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.688171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.698560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.706466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.739916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.748871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:03.748974] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:03.750425] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:03.750487] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668341, retry interval=1000000, ret=-4179) [2022-08-08 10:49:03.752903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.764743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.785063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.785171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:03.816305] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.819967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.820004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:03.845123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.845258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:03.935782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:03.946266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.959920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:03.980486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:03.997821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:04.029569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:04.049759] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:04.095010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:04.122357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:04.176643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:04.180166] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:04.198759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:04.198830] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:04.200205] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:04.200244] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668345, retry interval=1000000, ret=-4179) [2022-08-08 10:49:04.245890] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:04.249557] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:04.249601] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:04.269847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:04.429845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:04.756561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:04.775586] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:04.775889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:04.777333] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:04.777370] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668340, retry interval=1000000, ret=-4179) [2022-08-08 10:49:05.201295] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659926945201287, cur_ts_=1659926945173293, delta=27994) [2022-08-08 10:49:05.216940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2979-0-0] [lt=767] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:05.219455] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39062) [2022-08-08 10:49:05.220518] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=40947, time_dist: generate_cluster_version=40942) [2022-08-08 10:49:05.252893] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:05.256289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E24-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.256337] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E24-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.256352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E24-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.256365] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E24-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.256370] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E24-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.256388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E24-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.263059] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43557) [2022-08-08 10:49:05.263695] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=71944, time_dist: generate_cluster_version=71940) [2022-08-08 10:49:05.310575] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926945310562, cur_ts_=1659926945263645, delta=46917) [2022-08-08 10:49:05.348916] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=2904] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:05.348942] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:05.401446] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.401794] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.404160] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:05.404203] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668344, retry interval=1000000, ret=-4179) [2022-08-08 10:49:05.782155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.803023] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.803142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:05.804493] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:05.804562] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668339, retry interval=1000000, ret=-4179) [2022-08-08 10:49:06.033206] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17768) [2022-08-08 10:49:06.033488] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21626, time_dist: generate_cluster_version=21623) [2022-08-08 10:49:06.225789] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=726] [dc=0] clock out of order(cur_ts=1659926946225782, cur_ts_=1659926946181770, delta=44012) [2022-08-08 10:49:06.247301] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21697) [2022-08-08 10:49:06.253881] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33192, time_dist: do_cluster_heartbeat=10, generate_cluster_version=33176) [2022-08-08 10:49:06.355799] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=173] [dc=0] clock out of order(cur_ts=1659926946355790, cur_ts_=1659926946312546, delta=43244) [2022-08-08 10:49:06.366516] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:06.384869] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE297C-0-0] [lt=16] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659926946335459 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659926946383154 abs_expired_time=1659926948311147 stmt_timeout=1927993 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21584969 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659926946220547 start_version=-1 trans_version=1659926946335449 commit_version=0 stmt_start_time=1659926946181770 abs_expired_time=1659926976121106 stmt_timeout=29939336 abs_lock_wait_timeout=1659926976121106 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:2638503646927337938, inc:84067789, addr:"192.168.11.128:2882", t:1659926946181770} checksum_log_ts=9223372036854775807, version=1659926946335459) [2022-08-08 10:49:06.406429] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:06.406453] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:06.417318] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31411) [2022-08-08 10:49:06.447512] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15842) [2022-08-08 10:49:06.447926] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17195, time_dist: do_cluster_heartbeat=8, generate_cluster_version=17181) [2022-08-08 10:49:06.486571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:06.486623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:06.491373] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:06.491418] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668343, retry interval=1000000, ret=-4179) [2022-08-08 10:49:06.759257] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926946759251, cur_ts_=1659926946724870, delta=34381) [2022-08-08 10:49:06.807399] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E27-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:06.807540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E27-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:06.834391] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:06.834429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:06.836158] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:06.836197] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668338, retry interval=1000000, ret=-4179) [2022-08-08 10:49:06.941344] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659926946941337, cur_ts_=1659926946917795, delta=23542) [2022-08-08 10:49:07.417015] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=9] [dc=0] [trans sampling] (TRACE=begin_ts=1659926947416143 2022-08-08 02:49:07.416143|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:12383711477493381873, inc:84067832, addr:{ip:"192.168.11.128", port:2882}, t:1659926947414000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=37 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A56E0-0-0, start_time:1659926947416024, snapshot:1659926947365426|[end_task] u=140 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659926947416333|[calc_checksum_by_commit] u=31 ret:0, checksum:855212473, checksum_log_ts:9223372036854775807|[submit_log] u=56 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=508 trans_version:1659926947416359, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8298558, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=834) [2022-08-08 10:49:07.429604] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:07.432722] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:07.432764] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:07.494063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C461D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:07.494455] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C461D-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:07.517809] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:07.518044] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=173] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:07.523038] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:07.523478] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=344] [dc=0] task process failed, start retry(max retry time=9223372036854668342, retry interval=1000000, ret=-4179) [2022-08-08 10:49:07.882991] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:07.883044] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:07.888597] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:07.888641] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668337, retry interval=1000000, ret=-4179) [2022-08-08 10:49:08.039331] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17379) [2022-08-08 10:49:08.039601] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18597, time_dist: do_cluster_heartbeat=10, generate_cluster_version=18585) [2022-08-08 10:49:08.435659] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:08.438971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:08.439032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:08.539555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:08.539715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:08.561577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:08.561600] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:08.564323] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:08.564377] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668341, retry interval=1000000, ret=-4179) [2022-08-08 10:49:08.895289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=169] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:08.895307] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:08.934528] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] clock out of order(cur_ts=1659926948934518, cur_ts_=1659926948911079, delta=23439) [2022-08-08 10:49:08.936345] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:08.936380] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668336, retry interval=1000000, ret=-4179) [2022-08-08 10:49:09.377588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:09.397753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:09.448627] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:09.456231] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:09.456269] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:09.512563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:09.591222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:09.591250] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:09.593424] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:09.593463] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668340, retry interval=1000000, ret=-4179) [2022-08-08 10:49:09.600450] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:49:09.600468] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:49:09.710686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:09.723375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:09.767037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:09.767157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:09.767182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:09.779927] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:09.816824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:09.836056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:09.852370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:09.882635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:09.884815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:09.887137] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14567424}) [2022-08-08 10:49:09.894976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:09.965718] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:09.966326] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854668335, retry interval=1000000, ret=-4179) [2022-08-08 10:49:09.978791] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11924) [2022-08-08 10:49:09.999120] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:10.051681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:10.091516] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926948188838, end_time=1659926950091504, elapsed_time=1902666, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:10.214785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:10.263037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:10.324927] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:10.457905] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:10.463021] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:10.463064] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:10.630776] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:10.630843] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668339, retry interval=1000000, ret=-4179) [2022-08-08 10:49:10.997887] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:10.999642] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=1390] [dc=0] task process failed, start retry(max retry time=9223372036854668334, retry interval=1000000, ret=-4179) [2022-08-08 10:49:11.465748] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:11.469199] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:11.469716] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=502] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:11.638262] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:11.638313] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:11.664669] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:11.664785] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] task process failed, start retry(max retry time=9223372036854668338, retry interval=1000000, ret=-4179) [2022-08-08 10:49:12.002023] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:12.002041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:12.002046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:12.023132] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:12.023180] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668333, retry interval=1000000, ret=-4179) [2022-08-08 10:49:12.474894] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:12.478073] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:12.478097] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:12.617341] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=1143] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:12.690408] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659926952690401, cur_ts_=1659926952660966, delta=29435) [2022-08-08 10:49:12.710920] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21328) [2022-08-08 10:49:12.711488] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23678, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23665) [2022-08-08 10:49:12.720023] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32078, time_dist: do_cluster_heartbeat=7, generate_cluster_version=32068) [2022-08-08 10:49:12.729268] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:12.729296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:12.730537] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:12.730649] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668337, retry interval=1000000, ret=-4179) [2022-08-08 10:49:12.993832] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926952993824, cur_ts_=1659926952961383, delta=32441) [2022-08-08 10:49:13.082128] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:13.082171] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668332, retry interval=1000000, ret=-4179) [2022-08-08 10:49:13.096502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:13.199495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.207484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.249874] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:13.255346] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.377650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.388012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.393771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:13.419284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.419508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:13.420300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.453964] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659926953453113 2022-08-08 02:49:13.453113|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:1373909945517366375, inc:84068063, addr:{ip:"192.168.11.128", port:2882}, t:1659926953451000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A56FE-0-0, start_time:1659926953452995, snapshot:1659926953416304|[end_task] u=122 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926953453252|[calc_checksum_by_commit] u=42 ret:0, checksum:1571533567, checksum_log_ts:9223372036854775807|[submit_log] u=25 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=516 trans_version:1659926953453295, uref:1073741825|[exiting] u=20 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=48 ret:0, log_type:64, log_id:8298588, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=813) [2022-08-08 10:49:13.461047] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:13.480726] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:13.484498] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:13.484543] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:13.535140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.550904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.573047] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:13.584680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.587263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.591685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.603493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:13.608116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:13.645035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.645216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.651212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.676125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=1173] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.684403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:13.684452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.691250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.696521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=294] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.701612] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=369] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.701678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.701737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:13.701746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.705364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.724335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.729400] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.736128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:13.757202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:13.757431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:13.758836] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:13.758886] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668336, retry interval=1000000, ret=-4179) [2022-08-08 10:49:13.761290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.771547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.786490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.807363] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:13.807521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:13.839199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.840025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.843017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:13.862334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.870338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:13.959028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:13.969002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:13.977604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:14.000125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:14.018282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:14.047339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:14.072131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:14.087686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47AA-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:14.090170] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:14.103992] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10407, time_dist: generate_cluster_version=10401) [2022-08-08 10:49:14.112934] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:14.117518] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:14.117789] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=242] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:14.119517] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:14.119635] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668331, retry interval=1000000, ret=-4179) [2022-08-08 10:49:14.140108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:14.193107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:14.290464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:14.455628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:14.455844] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3264] [dc=0] clock out of order(cur_ts=1659926954455841, cur_ts_=1659926954424963, delta=30878) [2022-08-08 10:49:14.487277] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:14.492468] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:14.492523] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:14.618983] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659926954618975, cur_ts_=1659926954598228, delta=20747) [2022-08-08 10:49:14.787047] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:14.787069] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:14.788743] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:14.788838] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668335, retry interval=1000000, ret=-4179) [2022-08-08 10:49:14.991076] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:49:15.146531] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:15.146672] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] task process failed, start retry(max retry time=9223372036854668330, retry interval=1000000, ret=-4179) [2022-08-08 10:49:15.495148] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:15.498460] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:15.498498] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:15.546538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2997-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:15.546564] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2997-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:15.546708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2997-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:15.546883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2997-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:15.546938] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2997-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:15.547160] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2997-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:15.794037] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:15.794131] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:15.813003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:15.815323] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=210] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:15.815435] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668334, retry interval=1000000, ret=-4179) [2022-08-08 10:49:16.171545] WARN [LIB] ~ObTimeGuard (utility.h:788) [10915][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard 'rpc post' cost too much time, used=18685, time_dist: 5278,13386,2,13) [2022-08-08 10:49:16.197875] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926956197866, cur_ts_=1659926956170405, delta=27461) [2022-08-08 10:49:16.197899] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25851) [2022-08-08 10:49:16.226542] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:16.226653] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:16.228261] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:16.228393] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668329, retry interval=1000000, ret=-4179) [2022-08-08 10:49:16.493657] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659926956493651, cur_ts_=1659926956468843, delta=24808) [2022-08-08 10:49:16.494937] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659926956493309 2022-08-08 02:49:16.493309|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:2563565160803986084, inc:84068169, addr:{ip:"192.168.11.128", port:2882}, t:1659926956468000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A570D-0-0, start_time:1659926956493195, snapshot:1659926956372164|[end_task] u=120 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926956493446|[calc_checksum_by_commit] u=20 ret:0, checksum:1239132744, checksum_log_ts:9223372036854775807|[submit_log] u=25 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1356 trans_version:1659926956493467, uref:1073741825|[exiting] u=25 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8298603, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1590) [2022-08-08 10:49:16.501845] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:16.507237] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:16.507275] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:16.822961] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:16.823011] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:16.840513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:16.840578] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:16.842819] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:16.842858] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668333, retry interval=1000000, ret=-4179) [2022-08-08 10:49:17.243186] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:17.263526] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:17.265596] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:17.265637] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668328, retry interval=1000000, ret=-4179) [2022-08-08 10:49:17.509509] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:17.513134] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:17.513187] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:17.867036] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:17.867230] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:17.868516] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:17.868565] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668332, retry interval=1000000, ret=-4179) [2022-08-08 10:49:18.223991] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926958223981, cur_ts_=1659926958188496, delta=35485) [2022-08-08 10:49:18.325698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:18.326009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:18.327606] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:18.327672] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668327, retry interval=1000000, ret=-4179) [2022-08-08 10:49:18.380053] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659926958380045, cur_ts_=1659926958353177, delta=26868) [2022-08-08 10:49:18.382931] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27117) [2022-08-08 10:49:18.383115] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28736, time_dist: generate_cluster_version=28734) [2022-08-08 10:49:18.515800] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:18.520052] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:18.520098] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:18.880727] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926958880720, cur_ts_=1659926958858516, delta=22204) [2022-08-08 10:49:18.897038] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15509) [2022-08-08 10:49:18.898586] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18668, time_dist: do_cluster_heartbeat=11, generate_cluster_version=18652) [2022-08-08 10:49:18.898922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:18.898981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:18.908683] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11606) [2022-08-08 10:49:18.916403] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24411, time_dist: generate_cluster_version=24408) [2022-08-08 10:49:18.982810] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:18.982870] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668331, retry interval=1000000, ret=-4179) [2022-08-08 10:49:19.137943] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11285) [2022-08-08 10:49:19.138187] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12272, time_dist: generate_cluster_version=12267, cluster_service_self_check=3) [2022-08-08 10:49:19.295487] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926959295480, cur_ts_=1659926959270378, delta=25102) [2022-08-08 10:49:19.356617] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:19.356920] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668326, retry interval=1000000, ret=-4179) [2022-08-08 10:49:19.399674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:19.418060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:19.524181] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:19.529984] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:19.530139] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:19.533157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:19.602447] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:49:19.602473] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:49:19.727833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:19.748518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:19.784081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:19.791041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:19.791194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:19.804064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:19.832358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:19.856981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:19.874489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:19.905911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:19.906206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=222] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:19.926081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:19.932850] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15831) [2022-08-08 10:49:19.934151] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14571776}) [2022-08-08 10:49:19.934269] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18665, time_dist: do_cluster_heartbeat=12, generate_cluster_version=18650) [2022-08-08 10:49:20.013568] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:20.014975] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:20.015039] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668330, retry interval=1000000, ret=-4179) [2022-08-08 10:49:20.022319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:20.067393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:20.119051] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926958196236, end_time=1659926960119040, elapsed_time=1922804, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:20.230983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:20.284587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:20.349012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:20.386598] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=12070) [2022-08-08 10:49:20.393456] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:20.394249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:20.395360] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=1620] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21534, time_dist: do_cluster_heartbeat=9, generate_cluster_version=21522) [2022-08-08 10:49:20.396189] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:20.396253] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668325, retry interval=1000000, ret=-4179) [2022-08-08 10:49:20.532570] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:20.537352] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:20.537499] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:21.018094] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4676-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:21.018999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4676-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:21.019158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4676-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:21.046240] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:21.046275] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668329, retry interval=1000000, ret=-4179) [2022-08-08 10:49:21.418184] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:21.418234] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:21.419966] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:21.420030] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668324, retry interval=1000000, ret=-4179) [2022-08-08 10:49:21.540945] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=696] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:21.544079] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:21.544116] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:22.073512] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:22.073601] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:22.074986] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:22.075026] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668328, retry interval=1000000, ret=-4179) [2022-08-08 10:49:22.422764] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47B6-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:22.422846] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47B6-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:22.445500] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:22.445562] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668323, retry interval=1000000, ret=-4179) [2022-08-08 10:49:22.538585] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926962538575, cur_ts_=1659926962511093, delta=27482) [2022-08-08 10:49:22.541905] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659926962539670 2022-08-08 02:49:22.539670|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:1250111528612564569, inc:84068398, addr:{ip:"192.168.11.128", port:2882}, t:1659926962511000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB4F4-0-0, start_time:1659926962539551, snapshot:1659926962457076|[end_task] u=131 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=75 stc:1659926962539885|[calc_checksum_by_commit] u=23 ret:0, checksum:2344402267, checksum_log_ts:9223372036854775807|[submit_log] u=34 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1884 trans_version:1659926962511093, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313221, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=5 arg1:140085233538864|total_timeu=2198) [2022-08-08 10:49:22.546375] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:22.549988] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:22.550014] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:23.054011] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659926963054007, cur_ts_=1659926963016571, delta=37436) [2022-08-08 10:49:23.101317] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:23.101340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:23.103122] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:23.103172] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668327, retry interval=1000000, ret=-4179) [2022-08-08 10:49:23.122048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=610] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:23.220136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.226047] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.279564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.313161] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:23.402458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.413114] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.414572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:23.434742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:23.437151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.437189] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.477612] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:23.485481] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:23.485524] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668322, retry interval=1000000, ret=-4179) [2022-08-08 10:49:23.554711] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:23.555019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.566518] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:23.566654] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:23.575899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.589119] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:23.609451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.609512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.610453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.620233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:23.632961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:23.640365] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29AF-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:23.640545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29AF-0-0] [lt=146] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:23.640910] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29AF-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:23.641064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29AF-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:23.641365] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:23.666113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.690244] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659926963690240, cur_ts_=1659926963664458, delta=25782) [2022-08-08 10:49:23.689654] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.701605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.703075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.703228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:23.710447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.712687] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.732057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.732473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.736051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:23.736194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.736229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.736305] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.742982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.746507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.777748] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.788046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.808869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.829008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:23.829381] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:23.856004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.865204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.866431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:23.885773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.885984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:23.927097] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12721) [2022-08-08 10:49:23.927298] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13651, time_dist: generate_cluster_version=13648) [2022-08-08 10:49:23.978614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:23.986973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:23.997435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:24.019424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=281] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:24.042546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=181] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:24.066280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:24.089073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:24.107121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47B9-0-0] [lt=88] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:24.129424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:24.133802] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:24.133843] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668326, retry interval=1000000, ret=-4179) [2022-08-08 10:49:24.160271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:24.216704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:24.314540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:24.481077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:24.513925] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:24.515465] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:24.515527] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668321, retry interval=1000000, ret=-4179) [2022-08-08 10:49:24.575992] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=190] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:24.582434] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:24.582539] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:25.136193] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C467C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:25.136371] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C467C-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:25.136458] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C467C-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:25.138779] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:25.138801] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:25.169999] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:25.170057] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668325, retry interval=1000000, ret=-4179) [2022-08-08 10:49:25.539555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:25.541244] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:25.541293] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668320, retry interval=1000000, ret=-4179) [2022-08-08 10:49:25.584795] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:25.589954] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:25.589991] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:26.176886] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:26.176996] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:26.198222] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:26.198262] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668324, retry interval=1000000, ret=-4179) [2022-08-08 10:49:26.569997] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:26.570199] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] task process failed, start retry(max retry time=9223372036854668319, retry interval=1000000, ret=-4179) [2022-08-08 10:49:26.592332] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:26.599395] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:26.599433] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:26.998857] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5978] [dc=0] clock out of order(cur_ts=1659926966998850, cur_ts_=1659926966972265, delta=26585) [2022-08-08 10:49:27.226808] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:27.226858] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668323, retry interval=1000000, ret=-4179) [2022-08-08 10:49:27.250730] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659926967250723, cur_ts_=1659926967226777, delta=23946) [2022-08-08 10:49:27.251131] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10526][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=23522) [2022-08-08 10:49:27.580267] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:27.599939] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:27.599995] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:27.601486] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:27.601546] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668318, retry interval=1000000, ret=-4179) [2022-08-08 10:49:27.602522] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:27.605851] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:27.605885] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:27.923453] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926967923443, cur_ts_=1659926967899793, delta=23650) [2022-08-08 10:49:28.265780] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:28.265851] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668322, retry interval=1000000, ret=-4179) [2022-08-08 10:49:28.562477] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659926968561470 2022-08-08 02:49:28.561470|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:3148188148858296273, inc:84068623, addr:{ip:"192.168.11.128", port:2882}, t:1659926968559000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5749-0-0, start_time:1659926968561321, snapshot:1659926968560157|[end_task] u=129 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926968561616|[calc_checksum_by_commit] u=19 ret:0, checksum:3689481450, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=639 trans_version:1659926968561637, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8298663, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=905) [2022-08-08 10:49:28.609130] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:28.617293] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:28.617324] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:28.662703] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659926968662698, cur_ts_=1659926968634839, delta=27859) [2022-08-08 10:49:28.694456] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:28.694496] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668317, retry interval=1000000, ret=-4179) [2022-08-08 10:49:28.800932] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12417) [2022-08-08 10:49:28.813782] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10631][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511628002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8313252, proposal_id:{time_to_usec:1659815620290075, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:46124168, size:567}, after_consume_timestamp:1659926968800690}, end_time=1659926968813717, begin_time=1659926968800690, cost_time=13027) [2022-08-08 10:49:28.814069] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26351, time_dist: generate_cluster_version=26348) [2022-08-08 10:49:28.815321] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14313) [2022-08-08 10:49:28.815572] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17739, time_dist: generate_cluster_version=17735) [2022-08-08 10:49:28.928389] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926968928384, cur_ts_=1659926968893419, delta=34965) [2022-08-08 10:49:28.996138] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11218, time_dist: do_cluster_heartbeat=6, generate_cluster_version=11207) [2022-08-08 10:49:29.273773] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4682-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:29.273807] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4682-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:29.273822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4682-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:29.317641] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926969317630, cur_ts_=1659926969297204, delta=20426) [2022-08-08 10:49:29.328088] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:29.328174] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668321, retry interval=1000000, ret=-4179) [2022-08-08 10:49:29.375998] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39560) [2022-08-08 10:49:29.403764] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17091) [2022-08-08 10:49:29.404379] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=791] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19322, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19310) [2022-08-08 10:49:29.406125] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20158, time_dist: do_cluster_heartbeat=13, generate_cluster_version=20141) [2022-08-08 10:49:29.417384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:29.436146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:29.551613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:29.600499] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:49:29.600534] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:49:29.635794] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:29.639490] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:29.639999] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=492] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:29.700593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.700617] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.700623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.700628] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.703348] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.703376] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.703381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29C1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:29.737848] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:29.737928] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] task process failed, start retry(max retry time=9223372036854668316, retry interval=1000000, ret=-4179) [2022-08-08 10:49:29.748850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:29.769530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:29.810136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:29.812424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:29.812490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:29.819999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:29.851295] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:29.881566] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:29.891998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:29.925997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:29.926255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=223] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:29.950792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:29.974956] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14576128}) [2022-08-08 10:49:29.979546] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22566) [2022-08-08 10:49:29.980041] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30223, time_dist: do_cluster_heartbeat=9, generate_cluster_version=30211) [2022-08-08 10:49:30.044217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:30.088076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:30.092309] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659926970092301, cur_ts_=1659926970060408, delta=31893) [2022-08-08 10:49:30.210977] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=468] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926968189951, end_time=1659926970210967, elapsed_time=2021016, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:30.251979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:30.305414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:30.355522] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:30.355571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:30.357088] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:30.357123] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668320, retry interval=1000000, ret=-4179) [2022-08-08 10:49:30.367414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:30.644252] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:30.648297] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=188] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:30.648334] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:30.761491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:30.761685] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:30.763130] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:30.763219] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668315, retry interval=1000000, ret=-4179) [2022-08-08 10:49:31.384496] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:31.386464] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:31.386506] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668319, retry interval=1000000, ret=-4179) [2022-08-08 10:49:31.651104] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:31.654791] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:31.654838] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:31.762138] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659926971762092, cur_ts_=1659926971725606, delta=36486) [2022-08-08 10:49:31.784294] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:31.784344] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668314, retry interval=1000000, ret=-4179) [2022-08-08 10:49:32.293706] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659926972293694, cur_ts_=1659926972271142, delta=22552) [2022-08-08 10:49:32.389122] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47C5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:32.407349] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:32.407513] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668318, retry interval=1000000, ret=-4179) [2022-08-08 10:49:32.617768] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:32.659263] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:32.664385] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:32.664422] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:32.790970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:32.808383] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:32.808444] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668313, retry interval=1000000, ret=-4179) [2022-08-08 10:49:33.140219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:33.186196] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=12098) [2022-08-08 10:49:33.187633] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14295, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14281) [2022-08-08 10:49:33.236709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.251756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.299384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.381896] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27586) [2022-08-08 10:49:33.382265] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28819, time_dist: do_cluster_heartbeat=28, generate_cluster_version=28787) [2022-08-08 10:49:33.387391] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:33.411089] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4688-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:33.413854] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:33.421786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.433086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.433179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:33.433710] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:33.433767] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668317, retry interval=1000000, ret=-4179) [2022-08-08 10:49:33.434502] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11813) [2022-08-08 10:49:33.434966] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13069, time_dist: generate_cluster_version=13066) [2022-08-08 10:49:33.457756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:33.458099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.458424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.500832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:33.572108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.595552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.605832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:33.625266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.626667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.626955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.635725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:33.658061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:33.667335] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:33.670655] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:33.670791] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:33.687325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=24161] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.711002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=7235] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.718448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.718478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:33.723865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.728925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.734334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=372] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.749205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.751189] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:33.751221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.753104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.754141] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.761640] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.764634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.765529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.802849] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.816875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.827102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.829916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:33.829978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:33.831436] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:33.831498] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668312, retry interval=1000000, ret=-4179) [2022-08-08 10:49:33.844503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:33.851840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:33.876127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.881697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.881890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:33.902626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:33.904405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:33.999655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=158] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:34.006370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.019569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:34.063779] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926974063768, cur_ts_=1659926974029962, delta=33806) [2022-08-08 10:49:34.075175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.075292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:34.081213] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=49654) [2022-08-08 10:49:34.081279] WARN [STORAGE.TRANS] submit_log_impl_ (ob_trans_part_ctx.cpp:6697) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] submitting log costs too much time(ret=0, ret="OB_SUCCESS", timeguard=time guard 'submit_log_impl' cost too much time, used=49750, time_dist: 1,19,49723,3, trans_id={hash:512462508021051072, inc:84068847, addr:"192.168.11.128:2882", t:1659926974029962}, partition={tid:1099511628002, partition_id:0, part_cnt:0}, state={prepare_version:1659926974029962, state:0}, real_log_type=64, log_size=463) [2022-08-08 10:49:34.081289] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard 'submit_log_impl' cost too much time, used=49776, time_dist: 1,19,49723,3) [2022-08-08 10:49:34.081852] TRACE [TRACE]ob_trans_ctx.cpp:487 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] [trans warn] (TRACE=begin_ts=1659926974031281 2022-08-08 02:49:34.031281|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:512462508021051072, inc:84068847, addr:{ip:"192.168.11.128", port:2882}, t:1659926974029962}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB52E-0-0, start_time:1659926974031171, snapshot:1659926974024996|[end_task] u=209 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926974031507|[calc_checksum_by_commit] u=18 ret:0, checksum:933059708, checksum_log_ts:9223372036854775807|[submit_log] u=49728 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=38 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=468 trans_version:1659926974029962, uref:1073741825|[exiting] u=30 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313279, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=50534) [2022-08-08 10:49:34.081924] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=51161, time_dist: generate_cluster_version=51159) [2022-08-08 10:49:34.085172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.112605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.150499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.180168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:34.244952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:34.256467] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10535) [2022-08-08 10:49:34.263202] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18107, time_dist: do_cluster_heartbeat=43, generate_cluster_version=18060) [2022-08-08 10:49:34.341630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.436721] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E4E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.436911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E4E-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.437070] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E4E-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.437301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E4E-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.437425] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E4E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.437601] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.454308] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.454577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.456241] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:34.456292] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668316, retry interval=1000000, ret=-4179) [2022-08-08 10:49:34.496667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:34.673224] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:34.678285] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:34.678324] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:34.833827] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47C8-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.852553] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:34.853773] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:34.853824] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668311, retry interval=1000000, ret=-4179) [2022-08-08 10:49:35.026498] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:49:35.486761] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21169) [2022-08-08 10:49:35.486959] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22120, time_dist: generate_cluster_version=22118) [2022-08-08 10:49:35.498554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:35.500097] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:35.500157] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668315, retry interval=1000000, ret=-4179) [2022-08-08 10:49:35.681159] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:35.684975] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:35.685082] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:35.875701] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:35.875940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:35.877203] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:35.877256] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668310, retry interval=1000000, ret=-4179) [2022-08-08 10:49:36.502158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.502246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.502364] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.503679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=1212] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.504978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.505006] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.505021] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.505034] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.505039] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.505404] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.505482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47CB-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.526801] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.527085] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=257] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.528511] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:36.528547] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668314, retry interval=1000000, ret=-4179) [2022-08-08 10:49:36.687775] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:36.694326] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:36.694366] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:36.901817] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.901976] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:36.903328] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:36.903391] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668309, retry interval=1000000, ret=-4179) [2022-08-08 10:49:37.061495] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15003) [2022-08-08 10:49:37.061756] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16065, time_dist: generate_cluster_version=16062) [2022-08-08 10:49:37.531149] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C468E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.531323] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C468E-0-0] [lt=67] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.559315] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:37.559354] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668313, retry interval=1000000, ret=-4179) [2022-08-08 10:49:37.697024] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:37.700412] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:37.700537] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:37.750376] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659926977750369, cur_ts_=1659926977717434, delta=32935) [2022-08-08 10:49:37.906643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C464A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.906978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C464A-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.909315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.909384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.931870] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:37.933532] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:37.933623] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668308, retry interval=1000000, ret=-4179) [2022-08-08 10:49:38.635236] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:38.635307] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668312, retry interval=1000000, ret=-4179) [2022-08-08 10:49:38.703258] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:38.706537] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:38.706583] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:38.941561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:38.944939] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3176] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:38.977052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:38.977273] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:38.977859] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:38.977911] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668307, retry interval=1000000, ret=-4179) [2022-08-08 10:49:39.192220] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659926979192215, cur_ts_=1659926979161202, delta=31013) [2022-08-08 10:49:39.284621] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13370, time_dist: do_cluster_heartbeat=12, generate_cluster_version=13354) [2022-08-08 10:49:39.437492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:39.458784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:39.576509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:39.603922] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:49:39.603962] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:49:39.658743] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659926979658736, cur_ts_=1659926979635423, delta=23313) [2022-08-08 10:49:39.703201] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:39.703238] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668311, retry interval=1000000, ret=-4179) [2022-08-08 10:49:39.716635] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:39.719896] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:39.719950] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:39.776034] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:39.789122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:39.819388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29DF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.819413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29DF-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.819914] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29DF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.820080] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29DF-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.820345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.820389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.820406] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.820422] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:39.832019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:39.832333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:39.836373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:39.842319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:39.873115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=454] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:39.910731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:39.919191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=509] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:39.950264] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:39.963472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:39.978231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:40.017306] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:40.017344] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668306, retry interval=1000000, ret=-4179) [2022-08-08 10:49:40.022749] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14580544}) [2022-08-08 10:49:40.065409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:40.104252] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:40.232000] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926978190521, end_time=1659926980231986, elapsed_time=2041465, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:40.276166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:40.327208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:40.389755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:40.633016] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=20] [dc=0] [trans sampling] (TRACE=begin_ts=1659926980632022 2022-08-08 02:49:40.632022|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:4095626342973564216, inc:84069079, addr:{ip:"192.168.11.128", port:2882}, t:1659926980631000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB54F-0-0, start_time:1659926980631868, snapshot:1659926980608933|[end_task] u=225 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926980632264|[calc_checksum_by_commit] u=18 ret:0, checksum:3520170584, checksum_log_ts:9223372036854775807|[submit_log] u=43 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=601 trans_version:1659926980631475, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313312, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=952) [2022-08-08 10:49:40.706295] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47D1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:40.724161] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:40.728466] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:40.728503] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:40.732549] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:40.732641] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668310, retry interval=1000000, ret=-4179) [2022-08-08 10:49:40.930220] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926980930209, cur_ts_=1659926980910009, delta=20200) [2022-08-08 10:49:41.065499] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33458) [2022-08-08 10:49:41.065783] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35160, time_dist: do_cluster_heartbeat=10, generate_cluster_version=35147) [2022-08-08 10:49:41.072547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:41.072578] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:41.076160] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37608, time_dist: do_cluster_heartbeat=13, generate_cluster_version=37589) [2022-08-08 10:49:41.151713] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:41.152865] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] task process failed, start retry(max retry time=9223372036854668305, retry interval=1000000, ret=-4179) [2022-08-08 10:49:41.168389] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=48925, time_dist: generate_cluster_version=48920) [2022-08-08 10:49:41.235946] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659926981235938, cur_ts_=1659926981174091, delta=61847) [2022-08-08 10:49:41.287000] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=146122) [2022-08-08 10:49:41.287029] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=9, io_time=146145, file_id=237, flush_start_offset=46239532, file_offset=46240152) [2022-08-08 10:49:41.289984] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:9084846459769533945, inc:84069108, addr:"192.168.11.128:2882", t:1659926981123779}, log_id=8298727, used_time=149052) [2022-08-08 10:49:41.290469] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=150368, time_dist: generate_cluster_version=150365) [2022-08-08 10:49:41.329646] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42599) [2022-08-08 10:49:41.329734] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] transaction log sync use too much time(partition={tid:1099511627916, partition_id:0, part_cnt:0}, trans_id={hash:9722568179302463377, inc:84069109, addr:"192.168.11.128:2882", t:1659926981168005}, log_id=986908, used_time=160167) [2022-08-08 10:49:41.413934] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=127995, time_dist: do_cluster_heartbeat=10, generate_cluster_version=127977) [2022-08-08 10:49:41.418557] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] slow flush(flush_time=88885, io_time=42626, file_id=237, flush_start_offset=46240152, file_offset=46241408) [2022-08-08 10:49:41.451860] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926981451854, cur_ts_=1659926981275051, delta=176803) [2022-08-08 10:49:41.517686] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=99102) [2022-08-08 10:49:41.523891] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=106685, time_dist: do_cluster_heartbeat=12, generate_cluster_version=106668) [2022-08-08 10:49:41.526268] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=53832, time_dist: do_cluster_heartbeat=16, generate_cluster_version=53812) [2022-08-08 10:49:41.730956] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:41.734776] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:41.735774] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=982] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:41.760593] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:41.760657] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668309, retry interval=1000000, ret=-4179) [2022-08-08 10:49:41.899801] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926981899793, cur_ts_=1659926981841655, delta=58138) [2022-08-08 10:49:41.920698] WARN [COMMON] ObBucketTryRLockAllGuard (ob_bucket_lock.h:240) [10914][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to try read lock all buckets(ret=-4023) [2022-08-08 10:49:41.920722] WARN [STORAGE] get_create_schema_version (ob_pg_storage.cpp:1484) [10914][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get max pg schema version rdlock error(ret=-4023, pkey_={tid:1099511627956, partition_id:2, part_cnt:0}) [2022-08-08 10:49:41.976136] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13939, time_dist: do_cluster_heartbeat=781, generate_cluster_version=13154) [2022-08-08 10:49:42.020231] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926982020222, cur_ts_=1659926981983666, delta=36556) [2022-08-08 10:49:42.113685] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=108903) [2022-08-08 10:49:42.116079] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=2367, io_time=108927, file_id=237, flush_start_offset=46246422, file_offset=46247044) [2022-08-08 10:49:42.118812] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:12943818557985230604, inc:84069143, addr:"192.168.11.128:2882", t:1659926981983666}, log_id=8298731, used_time=114290) [2022-08-08 10:49:42.120571] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=7861] [dc=0] clock out of order(cur_ts=1659926982120567, cur_ts_=1659926982048426, delta=72141) [2022-08-08 10:49:42.121218] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=117881, time_dist: do_cluster_heartbeat=9, generate_cluster_version=117869) [2022-08-08 10:49:42.160009] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43899) [2022-08-08 10:49:42.170091] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=79453, time_dist: do_cluster_heartbeat=40, generate_cluster_version=79409) [2022-08-08 10:49:42.220993] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659926982220985, cur_ts_=1659926982186437, delta=34548) [2022-08-08 10:49:42.272856] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8425] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=86396) [2022-08-08 10:49:42.273387] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=93117, time_dist: do_cluster_heartbeat=10, generate_cluster_version=93104) [2022-08-08 10:49:42.308197] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23989) [2022-08-08 10:49:42.311866] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28473, time_dist: do_cluster_heartbeat=10, generate_cluster_version=28458) [2022-08-08 10:49:42.325059] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:42.325761] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] task process failed, start retry(max retry time=9223372036854668304, retry interval=1000000, ret=-4179) [2022-08-08 10:49:42.341477] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15209, time_dist: do_cluster_heartbeat=10, generate_cluster_version=15195) [2022-08-08 10:49:42.428159] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25486) [2022-08-08 10:49:42.428493] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28309, time_dist: generate_cluster_version=28306) [2022-08-08 10:49:42.511918] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17639] [dc=0] clock out of order(cur_ts=1659926982511908, cur_ts_=1659926982482870, delta=29038) [2022-08-08 10:49:42.678316] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1170] [dc=0] clock out of order(cur_ts=1659926982678309, cur_ts_=1659926982657866, delta=20443) [2022-08-08 10:49:42.741732] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:42.745286] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:42.745438] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:42.763425] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E5A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:42.763749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E5A-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:42.791120] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:42.791168] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668308, retry interval=1000000, ret=-4179) [2022-08-08 10:49:43.157951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:43.262034] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:43.270597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.315854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.354951] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:43.355009] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668303, retry interval=1000000, ret=-4179) [2022-08-08 10:49:43.467228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:43.467356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=359] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.467447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.469916] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:43.477410] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:43.477511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.481954] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659926983481944, cur_ts_=1659926983449488, delta=32456) [2022-08-08 10:49:43.483932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.523200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:43.536400] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21608) [2022-08-08 10:49:43.541485] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15051, time_dist: generate_cluster_version=15046) [2022-08-08 10:49:43.541545] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17225, time_dist: do_cluster_heartbeat=12, generate_cluster_version=17211) [2022-08-08 10:49:43.602053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.631788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:43.636960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:43.637286] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659926983637283, cur_ts_=1659926983612240, delta=25043) [2022-08-08 10:49:43.642270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.649103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.649187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.659320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:43.680352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:43.710974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.735727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.742341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.743158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.743200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:43.748623] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:43.751777] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:43.751839] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:43.753336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:43.756659] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:43.767051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:43.773959] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.773987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.774146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:43.774846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.777416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.780925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.784376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.811165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:43.811366] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:43.812951] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:43.813003] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668307, retry interval=1000000, ret=-4179) [2022-08-08 10:49:43.826178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.842577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.851992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.862274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:43.872840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:43.895907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.900033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:43.904407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:43.926324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:43.926968] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.016051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=1880] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:44.024885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.040917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=256] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:44.045198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29EB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:44.045230] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29EB-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:44.045244] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29EB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:44.045252] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29EB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:44.045339] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:44.091920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:44.099326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.102294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.130557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=266] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.172321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.196462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:44.262932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:44.312150] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22068) [2022-08-08 10:49:44.312330] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659926984312324, cur_ts_=1659926984289858, delta=22466) [2022-08-08 10:49:44.312394] WARN [LIB] ~ObTimeGuard (utility.h:788) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard 'rpc post' cost too much time, used=17902, time_dist: 17884,6,1,8) [2022-08-08 10:49:44.312460] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23181, time_dist: generate_cluster_version=23179) [2022-08-08 10:49:44.370764] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.377882] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:44.378051] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668302, retry interval=1000000, ret=-4179) [2022-08-08 10:49:44.521095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:44.758937] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:44.762818] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:44.762916] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:44.836947] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:44.836987] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668306, retry interval=1000000, ret=-4179) [2022-08-08 10:49:45.401539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:45.401774] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:45.403158] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:45.403283] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668301, retry interval=1000000, ret=-4179) [2022-08-08 10:49:45.766327] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:45.770220] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:45.770288] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:45.839612] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C469A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:45.856959] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:45.858372] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:45.858410] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668305, retry interval=1000000, ret=-4179) [2022-08-08 10:49:46.428134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:46.429872] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:46.429967] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] task process failed, start retry(max retry time=9223372036854668300, retry interval=1000000, ret=-4179) [2022-08-08 10:49:46.684397] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] [trans sampling] (TRACE=begin_ts=1659926986683613 2022-08-08 02:49:46.683613|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:9024583119130743683, inc:84069312, addr:{ip:"192.168.11.128", port:2882}, t:1659926986682000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB56F-0-0, start_time:1659926986683456, snapshot:1659926986600001|[end_task] u=127 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659926986683757|[calc_checksum_by_commit] u=18 ret:0, checksum:4114716567, checksum_log_ts:9223372036854775807|[submit_log] u=64 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=483 trans_version:1659926986683588, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313344, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233538864|total_timeu=744) [2022-08-08 10:49:46.772875] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:46.776199] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:46.776227] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:46.793755] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:49:46.883175] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:46.883248] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668304, retry interval=1000000, ret=-4179) [2022-08-08 10:49:46.978229] WARN [SQL] execute_get_plan (ob_sql.cpp:3081) [11024][0][YB42C0A80B80-0005E3CF9BC835E4-0-0] [lt=216] [dc=0] fail to get plan retry(ret=-5138) [2022-08-08 10:49:47.287799] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] clock out of order(cur_ts=1659926987287790, cur_ts_=1659926987253690, delta=34100) [2022-08-08 10:49:47.432868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47DA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:47.433073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47DA-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:47.433192] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47DA-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:47.454943] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:47.454980] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668299, retry interval=1000000, ret=-4179) [2022-08-08 10:49:47.778739] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:47.781977] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:47.782015] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:47.919127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:47.921458] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:47.921615] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] task process failed, start retry(max retry time=9223372036854668303, retry interval=1000000, ret=-4179) [2022-08-08 10:49:48.489522] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:48.490006] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=161] [dc=0] task process failed, start retry(max retry time=9223372036854668298, retry interval=1000000, ret=-4179) [2022-08-08 10:49:48.727196] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15615) [2022-08-08 10:49:48.727524] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16707, time_dist: generate_cluster_version=16704) [2022-08-08 10:49:48.728317] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13761, time_dist: generate_cluster_version=13758) [2022-08-08 10:49:48.784666] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:48.790846] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:48.790871] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:48.974751] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:48.974789] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668302, retry interval=1000000, ret=-4179) [2022-08-08 10:49:48.975264] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=290] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26894, time_dist: generate_cluster_version=26891) [2022-08-08 10:49:49.074865] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29FA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:49.074889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29FA-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:49.075083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29FA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:49.075397] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE29FA-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:49.075550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:49.460051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:49.474690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:49.516384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:49.517675] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:49.517722] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668297, retry interval=1000000, ret=-4179) [2022-08-08 10:49:49.599243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=2446] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:49.604330] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:49:49.604419] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:49:49.798987] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:49.805864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:49.807788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:49.829149] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:49.848704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:49.853065] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10622][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x6d3369c 0x3cd322f 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:49.853090] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14585024}) [2022-08-08 10:49:49.856052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:49.856945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:49.859331] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:49.890313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:49.910747] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659926989910738, cur_ts_=1659926989889920, delta=20818) [2022-08-08 10:49:49.933836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=8394] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:49.941434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:49.969845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:49.981356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:49.982728] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=13831) [2022-08-08 10:49:49.998112] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15715) [2022-08-08 10:49:49.998954] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30719, time_dist: generate_cluster_version=30716) [2022-08-08 10:49:50.004948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:50.017871] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=47959, time_dist: do_cluster_heartbeat=7, generate_cluster_version=47947) [2022-08-08 10:49:50.085917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:50.104983] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=263] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:50.105249] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] task process failed, start retry(max retry time=9223372036854668301, retry interval=1000000, ret=-4179) [2022-08-08 10:49:50.121138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:50.130166] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926988191130, end_time=1659926990130152, elapsed_time=1939022, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:50.300968] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:50.345724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:50.346260] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13255) [2022-08-08 10:49:50.348398] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16097, time_dist: generate_cluster_version=16093) [2022-08-08 10:49:50.407160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:50.540954] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:50.540997] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:50.542311] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:50.542360] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668296, retry interval=1000000, ret=-4179) [2022-08-08 10:49:50.858417] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:50.861996] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:50.862062] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:50.960883] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] clock out of order(cur_ts=1659926990960879, cur_ts_=1659926990932133, delta=28746) [2022-08-08 10:49:50.962279] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30038) [2022-08-08 10:49:50.962703] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=946] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32465, time_dist: generate_cluster_version=32462) [2022-08-08 10:49:51.108095] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E66-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.125603] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A00-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:51.125622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A00-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:51.125634] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A00-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:51.125643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A00-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:51.132112] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.132245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.133435] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:51.133484] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668300, retry interval=1000000, ret=-4179) [2022-08-08 10:49:51.547482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.547555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.569856] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.569942] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:51.574400] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:51.574518] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] task process failed, start retry(max retry time=9223372036854668295, retry interval=1000000, ret=-4179) [2022-08-08 10:49:51.866832] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:51.870117] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:51.870141] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:52.156381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:52.156555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:52.158426] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:52.158467] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668299, retry interval=1000000, ret=-4179) [2022-08-08 10:49:52.568290] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659926992568236, cur_ts_=1659926992545915, delta=22321) [2022-08-08 10:49:52.579690] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:52.600373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:52.600413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:52.601509] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:52.601568] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668294, retry interval=1000000, ret=-4179) [2022-08-08 10:49:52.618264] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:52.872819] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:52.876925] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:52.876964] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:53.178615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:53.184032] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:53.184093] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668298, retry interval=1000000, ret=-4179) [2022-08-08 10:49:53.282874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:53.292950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=3795] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.337862] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.482804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.488799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.491452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:53.496158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.501703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.501841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:53.543574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:53.592648] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:53.622204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.622224] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:53.623729] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:53.623792] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668293, retry interval=1000000, ret=-4179) [2022-08-08 10:49:53.654404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:53.658904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:53.661490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.671740] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.673723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.676441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:53.700547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:53.729869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.751249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.758035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.761434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:53.767618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.772414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:53.775493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:53.792321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:49:53.792456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.792530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:53.792568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.792692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.794805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.804216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.804250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=701] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.844043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.864838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.875095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.881073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:53.882075] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:53.885673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:53.885812] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:53.897043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:53.917841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.919229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:53.929480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:53.941925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:49:53.950146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.038256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:54.042042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.059697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:54.111218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:54.115379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.123107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.153049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.194577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.205243] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=154] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:54.205303] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668297, retry interval=1000000, ret=-4179) [2022-08-08 10:49:54.215069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:49:54.288471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:54.387087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.540237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:54.630733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:54.630791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:54.650055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:54.650304] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:54.651596] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:54.651650] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668292, retry interval=1000000, ret=-4179) [2022-08-08 10:49:54.759721] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=249] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10543, time_dist: do_cluster_heartbeat=9, generate_cluster_version=10530) [2022-08-08 10:49:54.809984] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=249] [dc=0] clock out of order(cur_ts=1659926994809974, cur_ts_=1659926994785841, delta=24133) [2022-08-08 10:49:54.888479] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:54.891892] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:54.891928] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:55.056954] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=2541] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:49:55.210084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:55.210116] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:55.243146] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:55.243596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:55.245460] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:55.245517] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668296, retry interval=1000000, ret=-4179) [2022-08-08 10:49:55.618282] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659926995618276, cur_ts_=1659926995583799, delta=34477) [2022-08-08 10:49:55.618745] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36079) [2022-08-08 10:49:55.618946] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37016, time_dist: generate_cluster_version=37015) [2022-08-08 10:49:55.673135] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:55.673232] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:55.674828] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=319] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:55.675028] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668291, retry interval=1000000, ret=-4179) [2022-08-08 10:49:55.896302] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:55.899442] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:55.899478] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:56.267469] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:56.267532] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668295, retry interval=1000000, ret=-4179) [2022-08-08 10:49:56.480920] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659926996480911, cur_ts_=1659926996449320, delta=31591) [2022-08-08 10:49:56.680019] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680157] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680328] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680416] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=71] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680534] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680681] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=74] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680769] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.680936] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.681104] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.681198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.681389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.681520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.681668] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.681981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.682027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E6F-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:56.710611] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:56.710686] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668290, retry interval=1000000, ret=-4179) [2022-08-08 10:49:56.905742] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:56.909000] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:56.909075] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:57.295380] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:57.295930] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668294, retry interval=1000000, ret=-4179) [2022-08-08 10:49:57.714883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4668-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:57.733503] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:57.733738] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:49:57.735285] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:57.735335] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668289, retry interval=1000000, ret=-4179) [2022-08-08 10:49:57.914635] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:57.919182] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=451] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:57.919295] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:58.179992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:58.180112] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:58.180289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:58.180437] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:58.326055] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:58.326105] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668293, retry interval=1000000, ret=-4179) [2022-08-08 10:49:58.454825] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27984) [2022-08-08 10:49:58.459508] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=33425, time_dist: generate_cluster_version=33421) [2022-08-08 10:49:58.747904] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659926998741378 2022-08-08 02:49:58.741378|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:14001736941492689983, inc:84069772, addr:{ip:"192.168.11.128", port:2882}, t:1659926998740000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB5AB-0-0, start_time:1659926998741258, snapshot:1659926998719701|[end_task] u=186 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659926998741583|[calc_checksum_by_commit] u=20 ret:0, checksum:2005972623, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=6175 trans_version:1659926998740668, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313404, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233538864|total_timeu=6459) [2022-08-08 10:49:58.763560] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:58.763691] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668288, retry interval=1000000, ret=-4179) [2022-08-08 10:49:58.781311] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15348) [2022-08-08 10:49:58.922059] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:58.935185] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=6581] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:58.935224] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:58.983026] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18348) [2022-08-08 10:49:58.983412] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19528, time_dist: generate_cluster_version=19526) [2022-08-08 10:49:59.190207] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A18-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:59.190229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A18-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:59.190406] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A18-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:59.190580] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A18-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:59.190856] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:49:59.254132] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659926999254122, cur_ts_=1659926999231519, delta=22603) [2022-08-08 10:49:59.353494] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:59.353530] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668292, retry interval=1000000, ret=-4179) [2022-08-08 10:49:59.374901] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20290) [2022-08-08 10:49:59.398680] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16999) [2022-08-08 10:49:59.403056] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23801, time_dist: generate_cluster_version=23797) [2022-08-08 10:49:59.435891] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11629) [2022-08-08 10:49:59.436145] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12641, time_dist: generate_cluster_version=12638) [2022-08-08 10:49:59.486327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:59.491922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:59.559264] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659926999559260, cur_ts_=1659926999530227, delta=29033) [2022-08-08 10:49:59.579345] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20625, time_dist: do_cluster_heartbeat=6, generate_cluster_version=20617) [2022-08-08 10:49:59.615936] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:59.619049] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:49:59.619070] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:49:59.692460] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659926999692450, cur_ts_=1659926999670274, delta=22176) [2022-08-08 10:49:59.774358] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14589312}) [2022-08-08 10:49:59.789777] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:49:59.789820] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668287, retry interval=1000000, ret=-4179) [2022-08-08 10:49:59.826906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=1852] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:59.826984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:49:59.866514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:59.878325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:59.878368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:59.878860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:59.914798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:49:59.937421] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:49:59.941197] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:49:59.941271] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:49:59.951813] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:59.964616] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659926998191995, end_time=1659926999964602, elapsed_time=1772607, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:49:59.966132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:49:59.990214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:49:59.997355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:00.021473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:00.104955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:00.142240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:00.324752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=259] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:00.368887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:00.384491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:00.384513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:00.393952] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=172] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:00.394020] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668291, retry interval=1000000, ret=-4179) [2022-08-08 10:50:00.407411] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10007, time_dist: generate_cluster_version=10001) [2022-08-08 10:50:00.409894] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10691, time_dist: do_cluster_heartbeat=7, generate_cluster_version=10680) [2022-08-08 10:50:00.427302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:00.809124] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16056, time_dist: generate_cluster_version=16054) [2022-08-08 10:50:00.811078] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:00.811264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:00.812645] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:00.812774] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] task process failed, start retry(max retry time=9223372036854668286, retry interval=1000000, ret=-4179) [2022-08-08 10:50:00.944711] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:00.948010] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:00.948049] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:01.077283] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927001077276, cur_ts_=1659927001056340, delta=20936) [2022-08-08 10:50:01.397202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46B1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:01.397449] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46B1-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:01.397776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46B1-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:01.397794] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46B1-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:01.422546] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=268] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:01.422821] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=202] [dc=0] task process failed, start retry(max retry time=9223372036854668290, retry interval=1000000, ret=-4179) [2022-08-08 10:50:01.543114] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927001543109, cur_ts_=1659927001517063, delta=26046) [2022-08-08 10:50:01.618806] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22834) [2022-08-08 10:50:01.619035] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24154, time_dist: do_cluster_heartbeat=10, generate_cluster_version=24142) [2022-08-08 10:50:01.620161] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24487, time_dist: do_cluster_heartbeat=7, generate_cluster_version=24477) [2022-08-08 10:50:01.775386] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927001774596 2022-08-08 02:50:01.774596|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:3065893525753117273, inc:84069889, addr:{ip:"192.168.11.128", port:2882}, t:1659927001774000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB5BA-0-0, start_time:1659927001774430, snapshot:1659927001747901|[end_task] u=133 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927001774747|[calc_checksum_by_commit] u=18 ret:0, checksum:3204860055, checksum_log_ts:9223372036854775807|[submit_log] u=54 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=492 trans_version:1659927001774511, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313419, logging:0, pending:0, uref:1|[end_trans_cb] u=15 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=751) [2022-08-08 10:50:01.842224] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:01.842274] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668285, retry interval=1000000, ret=-4179) [2022-08-08 10:50:01.951089] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:01.954438] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:01.954480] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:02.427049] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47EF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.427092] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47EF-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.427108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47EF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.427121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47EF-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.427134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C47EF-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.449686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.449706] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.451060] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:02.451109] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668289, retry interval=1000000, ret=-4179) [2022-08-08 10:50:02.844704] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46B4-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.861949] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:02.863085] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:02.863148] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668284, retry interval=1000000, ret=-4179) [2022-08-08 10:50:02.957109] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:02.960562] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:02.960606] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:03.194828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:03.301925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.311877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.353575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.476822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:03.476841] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:03.477859] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:03.477908] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668288, retry interval=1000000, ret=-4179) [2022-08-08 10:50:03.500104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.512107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:03.512509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.512609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.520353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.522686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:03.564142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:03.643984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.655838] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:03.675682] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.678843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.680263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:03.690291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.690355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.693719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:03.721770] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:03.752899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.773702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.774495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.783954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:03.786424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.793961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.794306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.814671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:03.814741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.814761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.814776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.814776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.814825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.825186] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.828530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.867051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.883650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.886690] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:03.886868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:03.888196] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:03.888248] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668283, retry interval=1000000, ret=-4179) [2022-08-08 10:50:03.893072] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.901124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:03.918908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:03.935757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.935768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:03.946218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.962274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=194] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:03.964926] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:03.967907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:03.968051] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:03.968084] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:04.060652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=243] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.060727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=879] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:04.076139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:04.092192] TRACE [TRACE]ob_trans_ctx.cpp:485 [10877][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] [slow trans] (TRACE=begin_ts=1659926903642188 2022-08-08 02:48:23.642188|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:14122346763008492130, inc:84066165, addr:{ip:"192.168.11.128", port:2882}, t:1659926903641018}, pkey:{tid:1100611139453778, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835DE-0-0, start_time:1659926903596428, snapshot:1659926903641783|[end_task] u=98304 ret:0, is_rollback:false, uref:1073741825|[start_task] u=5898322 ret:0, sql_no:8589934594, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835DF-0-0, start_time:1659926909630654, snapshot:1659926909590481|[end_task] u=17182 ret:0, is_rollback:false, uref:1073741825|[handle_timeout] u=4482451 ret:0, used:{click:[0, 0, 4]}, uref:1073741826|[handle_timeout] u=205874 ret:0, used:{click:[1, 0, 7]}, uref:1073741826|[handle_timeout] u=402745 ret:0, used:{click:[0, 1, 3]}, uref:1073741826|[handle_timeout] u=804049 ret:0, used:{click:[1, 0, 7]}, uref:1073741826|[handle_timeout] u=1604032 ret:0, used:{click:[1, 0, 5]}, uref:1073741826|[handle_timeout] u=3007871 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[start_task] u=1485995 ret:0, sql_no:12884901891, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835E0-0-0, start_time:1659926921648483, snapshot:1659926921634800|[end_task] u=169 ret:0, is_rollback:false, uref:1073741825|[handle_timeout] u=1516575 ret:0, used:{click:[1, 0, 8]}, uref:1073741826|[handle_timeout] u=3006953 ret:0, used:{click:[1, 0, 5]}, uref:1073741826|[handle_timeout] u=3005768 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=3002310 ret:0, used:{click:[2, 0, 6]}, uref:1073741826|[recover_dist_trans] u=110477 ret:0, uref:1073741825|[handle_message] u=23920475 ret:0, msg_type:14, request_id:-1, uref:1073741825|[handle_message] u=28100981 ret:0, msg_type:14, request_id:-1, uref:1073741825|[start_task] u=2700764 ret:0, sql_no:25769803782, uref:1073741825, snapshot:1659926987012420|[end_task] u=68 ret:0, is_rollback:false, uref:1073741825|[exiting] u=17078559 uref:1073741824, arg1:3221573953, arg2:0|[handle_timeout] u=3 ret:0, used:{click:[2, 0, 20]}, uref:1|[destroy] u=5 arg1:140085233547952|total_timeu=100449943) [2022-08-08 10:50:04.134621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:04.139564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.144188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.175770] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.217452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.228323] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A27-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:04.228413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A27-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:04.229150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:04.239298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:04.311684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:04.408963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.482345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.482408] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.506860] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.506921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.509521] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:04.509580] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668287, retry interval=1000000, ret=-4179) [2022-08-08 10:50:04.559233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:04.919344] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E7B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.919373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E7B-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.922853] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927004922843, cur_ts_=1659927004898546, delta=24297) [2022-08-08 10:50:04.923628] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.939059] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=599] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.939232] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:04.940632] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:04.940696] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668282, retry interval=1000000, ret=-4179) [2022-08-08 10:50:04.970601] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:04.976701] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:04.976725] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:05.531659] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:05.531697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:05.533033] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:05.533099] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668286, retry interval=1000000, ret=-4179) [2022-08-08 10:50:05.820410] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3661] [dc=0] clock out of order(cur_ts=1659927005820406, cur_ts_=1659927005794584, delta=25822) [2022-08-08 10:50:05.829346] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10709, time_dist: do_cluster_heartbeat=37, generate_cluster_version=10670) [2022-08-08 10:50:05.830680] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11107, time_dist: generate_cluster_version=11104) [2022-08-08 10:50:05.960630] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:05.960775] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:05.964135] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:05.964202] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668281, retry interval=1000000, ret=-4179) [2022-08-08 10:50:05.978974] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:05.982273] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:05.982311] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:06.242937] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:06.243255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:06.243392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:06.558494] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:06.558532] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668285, retry interval=1000000, ret=-4179) [2022-08-08 10:50:06.984898] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:06.985578] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:06.985612] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668280, retry interval=1000000, ret=-4179) [2022-08-08 10:50:06.988709] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:06.989126] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:07.027126] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17893, time_dist: generate_cluster_version=17888) [2022-08-08 10:50:07.582011] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:07.582053] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668284, retry interval=1000000, ret=-4179) [2022-08-08 10:50:07.826851] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927007825839 2022-08-08 02:50:07.825839|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17899252936323665040, inc:84070120, addr:{ip:"192.168.11.128", port:2882}, t:1659927007825000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A580F-0-0, start_time:1659927007825724, snapshot:1659927007771059|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927007825980|[calc_checksum_by_commit] u=22 ret:0, checksum:2128782376, checksum_log_ts:9223372036854775807|[submit_log] u=67 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=697 trans_version:1659927007826002, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8298861, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=973) [2022-08-08 10:50:07.996247] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:08.000805] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:08.000841] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:08.012309] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:08.013793] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:08.013844] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668279, retry interval=1000000, ret=-4179) [2022-08-08 10:50:08.399205] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=445] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10754, time_dist: generate_cluster_version=10750) [2022-08-08 10:50:08.453467] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927008453463, cur_ts_=1659927008426947, delta=26516) [2022-08-08 10:50:08.610846] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12944) [2022-08-08 10:50:08.611136] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14270, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14258) [2022-08-08 10:50:08.642230] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=475] [dc=0] clock out of order(cur_ts=1659927008642222, cur_ts_=1659927008610784, delta=31438) [2022-08-08 10:50:08.647791] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:08.647825] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668283, retry interval=1000000, ret=-4179) [2022-08-08 10:50:08.787475] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15158) [2022-08-08 10:50:08.787792] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16207, time_dist: generate_cluster_version=16204) [2022-08-08 10:50:09.007131] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:09.010319] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:09.010356] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:09.041533] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:09.041680] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668278, retry interval=1000000, ret=-4179) [2022-08-08 10:50:09.141252] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927009141243, cur_ts_=1659927009113813, delta=27430) [2022-08-08 10:50:09.509605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:09.509717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=235] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:09.616020] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:50:09.616055] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:50:09.632123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:09.685427] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:09.685639] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] task process failed, start retry(max retry time=9223372036854668282, retry interval=1000000, ret=-4179) [2022-08-08 10:50:09.778256] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14593696}) [2022-08-08 10:50:09.848355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:09.850003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:09.891469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:09.897404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:09.898852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:09.901397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:09.936566] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:09.979507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:09.982113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:09.982505] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927009982499, cur_ts_=1659927009955402, delta=27097) [2022-08-08 10:50:09.984609] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927008192473, end_time=1659927009984599, elapsed_time=1792126, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:50:10.010558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:10.011971] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:10.015328] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:10.015365] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:10.021515] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:10.043892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:10.062986] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:10.063101] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:10.064665] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:10.064721] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668277, retry interval=1000000, ret=-4179) [2022-08-08 10:50:10.136609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:10.136988] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927010136983, cur_ts_=1659927010111269, delta=25714) [2022-08-08 10:50:10.157858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:10.344889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:10.392469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:10.447524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=837] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:10.712547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:10.712584] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:10.714376] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:10.714452] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668281, retry interval=1000000, ret=-4179) [2022-08-08 10:50:10.827233] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927010826339 2022-08-08 02:50:10.826339|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17063726565265801149, inc:84070228, addr:{ip:"192.168.11.128", port:2882}, t:1659927010825000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=41 ret:0, left_time:10000000, uref:1073741825|[start_task] u=9 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A581E-0-0, start_time:1659927010826222, snapshot:1659927010794326|[end_task] u=138 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927010826532|[calc_checksum_by_commit] u=21 ret:0, checksum:1852015538, checksum_log_ts:9223372036854775807|[submit_log] u=65 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=534 trans_version:1659927010826554, uref:1073741825|[exiting] u=19 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8298876, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=857) [2022-08-08 10:50:11.017882] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:11.029092] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:11.029134] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:11.069071] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C0-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.069105] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C0-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.089521] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.089662] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.090730] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:11.090791] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668276, retry interval=1000000, ret=-4179) [2022-08-08 10:50:11.717095] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E84-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.717149] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E84-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.746558] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:11.747869] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:11.747959] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] task process failed, start retry(max retry time=9223372036854668280, retry interval=1000000, ret=-4179) [2022-08-08 10:50:12.032680] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:12.035760] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:12.035849] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:12.120490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:12.120732] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:12.122281] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:12.122347] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668275, retry interval=1000000, ret=-4179) [2022-08-08 10:50:12.274458] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A3F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.274571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A3F-0-0] [lt=91] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.274759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A3F-0-0] [lt=161] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.274963] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A3F-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.275206] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.275347] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.275409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:12.618545] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:12.783516] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:12.783563] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668279, retry interval=1000000, ret=-4179) [2022-08-08 10:50:13.040091] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:13.043612] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:13.043658] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:13.125075] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5E87-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:13.130999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:13.131083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:13.148849] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:13.150227] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:13.150290] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668274, retry interval=1000000, ret=-4179) [2022-08-08 10:50:13.215735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:13.324924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:13.336709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.377167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.521229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.536296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.536639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.537412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:13.540010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.546580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:13.580279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:13.660673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.695422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:13.695704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.704761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:13.706578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.708883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.717399] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:13.718144] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:13.737279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:13.768344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.798964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.800035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.802437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:13.804341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.811954] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:13.812028] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668278, retry interval=1000000, ret=-4179) [2022-08-08 10:50:13.812954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:13.815106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:13.833743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.835694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:13.835901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.836035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.836068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.837172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=154] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:13.846998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.847921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.869638] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927013868633 2022-08-08 02:50:13.868633|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:4651498770894541055, inc:84070350, addr:{ip:"192.168.11.128", port:2882}, t:1659927013867000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB5F6-0-0, start_time:1659927013868523, snapshot:1659927013826308|[end_task] u=120 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927013868769|[calc_checksum_by_commit] u=18 ret:0, checksum:4106215124, checksum_log_ts:9223372036854775807|[submit_log] u=26 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=744 trans_version:1659927013867928, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313479, logging:0, pending:0, uref:1|[end_trans_cb] u=24 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=967) [2022-08-08 10:50:13.888901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.909042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.916421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.918486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:13.939374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:13.960250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:13.960356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.963828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:13.984554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:13.993054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.045976] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:14.049519] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:14.049554] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:14.091644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.091697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:14.091725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:14.159277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:14.166985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.169567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.172357] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:14.172488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:14.174681] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:14.174720] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668273, retry interval=1000000, ret=-4179) [2022-08-08 10:50:14.200810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.242572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.264233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=148] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:14.284341] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A45-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.284572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A45-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.284775] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A45-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.284912] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A45-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.285293] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.285399] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.285538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:14.335771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:14.365120] WARN [SQL] execute_get_plan (ob_sql.cpp:3081) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=1051] [dc=0] fail to get plan retry(ret=-5138) [2022-08-08 10:50:14.431334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.444384] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26694) [2022-08-08 10:50:14.444733] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27728, time_dist: do_cluster_heartbeat=8, generate_cluster_version=27710) [2022-08-08 10:50:14.510969] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36230) [2022-08-08 10:50:14.511266] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37504, time_dist: generate_cluster_version=37502) [2022-08-08 10:50:14.546975] WARN [STORAGE.TRANS] start_stmt_sanity_check_ (ob_trans_service.cpp:10302) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=14] [dc=0] transaction is timeout(trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86690700, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E5-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927014365043, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927024295043, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}) [2022-08-08 10:50:14.547029] WARN [STORAGE.TRANS] start_stmt_v2 (ob_trans_service.cpp:9886) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=26] [dc=0] sanity check fail(ret=-6210, trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86690700, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E5-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927014365043, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927024295043, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}) [2022-08-08 10:50:14.547057] WARN [STORAGE.TRANS] start_stmt_v2 (ob_trans_service.cpp:9947) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=6] [dc=0] start statement failed(ret=-6210, trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86690700, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E5-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927014365043, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927024295043, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}, stmt_param=[tenant_id=1001, stmt_expired_time=1659927024295043, is_retry_sql=0, safe_weak_read_snapshot=0, weak_read_snapshot_source=0, weak_read_snapshot_source_str=UNKOWN, is_trx_elr=0, multi_tenant=0]) [2022-08-08 10:50:14.547104] WARN [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:1151) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=18] [dc=0] fail start stmt(ret=-6210, trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86690700, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E5-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927014365043, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927024295043, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}, pla={partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, out_pla={partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_remote=false, unreachable_partitions.count()=0) [2022-08-08 10:50:14.547131] WARN [SQL] start_stmt (ob_result_set.cpp:386) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=19] [dc=0] fail to start stmt(ret=-6210, participants={partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, physical_plan_->get_dependency_table()=[{table_id:1100611139453778, schema_version:1659926854292344, object_type:1, is_db_explicit:false, is_existed:true}]) [2022-08-08 10:50:14.547136] WARN [SQL.SESSION] do_open_plan (ob_result_set.cpp:660) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=4] [dc=0] fail start stmt(ret=-6210) [2022-08-08 10:50:14.547140] WARN [SQL.SESSION] sync_open (ob_result_set.cpp:140) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=3] [dc=0] fail to exec execute()(ret=-6210) [2022-08-08 10:50:14.547147] WARN [SERVER] test_and_save_retry_state (ob_query_retry_ctrl.cpp:527) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=2] [dc=0] do not need retry(client_ret=-6210, err=-6210, expected_stmt=true, THIS_WORKER.get_timeout_ts()=1659927024365043, retry_type_=0, result.get_stmt_type()=5, result.get_exec_context().need_change_timeout_ret()=true, session->get_retry_info().get_last_query_retry_err()=0) [2022-08-08 10:50:14.547171] WARN [SERVER] response_result (ob_sync_plan_driver.cpp:73) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=16] [dc=0] result set open failed, check if need retry(ret=-6210, cli_ret=-6210, retry_ctrl_.need_retry()=0) [2022-08-08 10:50:14.547179] WARN [SERVER] response_result (ob_sync_plan_driver.cpp:80) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=4] [dc=0] close result set fail(cret=-6210) [2022-08-08 10:50:14.571712] WARN [SERVER] do_process (obmp_query.cpp:775) [11024][0][YB42C0A80B80-0005E3CF9BC835E5-0-0] [lt=2] [dc=0] execute query fail(ret=-6210, timeout_timestamp=1659927024365043) [2022-08-08 10:50:14.579529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:14.613138] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927014613129, cur_ts_=1659927014591496, delta=21633) [2022-08-08 10:50:14.833197] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:14.833340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:14.837947] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:14.838067] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668277, retry interval=1000000, ret=-4179) [2022-08-08 10:50:15.053778] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:15.056971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:15.057008] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:15.082268] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:50:15.198642] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:15.198660] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:15.199876] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:15.199940] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668272, retry interval=1000000, ret=-4179) [2022-08-08 10:50:15.866718] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:15.866925] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] task process failed, start retry(max retry time=9223372036854668276, retry interval=1000000, ret=-4179) [2022-08-08 10:50:16.059351] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:16.062472] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:16.062515] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:16.121235] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927016121228, cur_ts_=1659927016092035, delta=29193) [2022-08-08 10:50:16.252944] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:16.253197] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668271, retry interval=1000000, ret=-4179) [2022-08-08 10:50:16.890910] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:16.890989] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:16.892993] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:16.893280] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668275, retry interval=1000000, ret=-4179) [2022-08-08 10:50:17.065489] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:17.071096] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:17.071137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:17.276957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.278750] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:17.278820] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668270, retry interval=1000000, ret=-4179) [2022-08-08 10:50:17.896873] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.897084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.897355] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.897467] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.897494] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.897845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.897983] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.898099] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46C9-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.898320] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.917784] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:17.919026] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:17.919096] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668274, retry interval=1000000, ret=-4179) [2022-08-08 10:50:18.073898] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:18.077460] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:18.077503] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:18.309609] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:18.310875] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:18.312088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.313929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=1817] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.313942] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.313947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.313984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.313996] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.314000] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A51-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:18.315006] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:18.315041] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668269, retry interval=1000000, ret=-4179) [2022-08-08 10:50:18.953928] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:18.953961] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668273, retry interval=1000000, ret=-4179) [2022-08-08 10:50:19.079935] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:19.084282] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:19.084306] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:19.354513] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:19.354574] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668268, retry interval=1000000, ret=-4179) [2022-08-08 10:50:19.530936] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:19.531015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:19.608415] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:50:19.608445] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:50:19.649128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:19.768269] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14598048}) [2022-08-08 10:50:19.865698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:19.867568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=2685] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:19.909227] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:19.917250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:19.918933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:19.920339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:19.945924] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927018193050, end_time=1659927019945910, elapsed_time=1752860, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:50:19.962592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:19.978855] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:19.978928] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:19.980538] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:19.980608] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854668272, retry interval=1000000, ret=-4179) [2022-08-08 10:50:19.995786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:20.001717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:20.032928] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:20.041856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:20.063700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:20.086152] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:20.089673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:20.089715] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:20.160420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:20.180842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:20.359448] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:20.359488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:20.368825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:20.384108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:20.384134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:20.385535] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:20.385711] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668267, retry interval=1000000, ret=-4179) [2022-08-08 10:50:20.417048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:20.470885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:20.982987] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C480C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:20.998698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:20.999944] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:20.999993] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668271, retry interval=1000000, ret=-4179) [2022-08-08 10:50:21.092594] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:21.098168] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:21.098208] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:21.435054] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:21.435112] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668266, retry interval=1000000, ret=-4179) [2022-08-08 10:50:22.002702] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46CF-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:22.002747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46CF-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:22.005946] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:22.023671] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:22.023718] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:22.025032] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:22.025093] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668270, retry interval=1000000, ret=-4179) [2022-08-08 10:50:22.119143] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:22.122639] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:22.122665] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:22.461913] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=220] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:22.461983] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668265, retry interval=1000000, ret=-4179) [2022-08-08 10:50:23.049118] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:23.049155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:23.056452] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:23.056639] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668269, retry interval=1000000, ret=-4179) [2022-08-08 10:50:23.125036] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:23.130198] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:23.130243] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:23.235679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:23.350312] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:23.360641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.394931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.481573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:23.481757] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:23.482973] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:23.483008] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668264, retry interval=1000000, ret=-4179) [2022-08-08 10:50:23.542822] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927023542816, cur_ts_=1659927023510412, delta=32404) [2022-08-08 10:50:23.543728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.552859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:23.553248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.557663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:23.563409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.565079] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:23.596346] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:23.677586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.717753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:23.718746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=198] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.724731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:23.729188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:23.730689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.741113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:23.762022] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:23.780436] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:23.792427] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.814574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.820496] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:23.820518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.820628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:23.831007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:23.834503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:23.851818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.853160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.854140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:23.854242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.855252] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:23.856708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:23.865727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=482] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.872986] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.906745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:23.927801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.939460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.943310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:23.960440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:23.979845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:23.981138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:23.984599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:24.001023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:24.011554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.079916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:24.080152] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:24.081639] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:24.081702] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668268, retry interval=1000000, ret=-4179) [2022-08-08 10:50:24.111085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:24.112771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.115397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:24.132918] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:24.136358] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:24.136397] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:24.179067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:24.186507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.190532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.219211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.264855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.288298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:24.360606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:24.375922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A63-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:24.376404] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A63-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:24.376642] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:24.377154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:24.451347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:24.486201] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46D2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:24.510632] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=125] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:24.512277] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:24.512394] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668263, retry interval=1000000, ret=-4179) [2022-08-08 10:50:24.604195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:25.105238] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:25.105276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:25.118380] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:25.118849] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854668267, retry interval=1000000, ret=-4179) [2022-08-08 10:50:25.138988] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:25.142092] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:25.142130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:25.520186] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:25.520227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:25.539637] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:25.539703] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668262, retry interval=1000000, ret=-4179) [2022-08-08 10:50:25.739028] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927025739021, cur_ts_=1659927025716135, delta=22886) [2022-08-08 10:50:26.081884] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18810) [2022-08-08 10:50:26.082183] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20284, time_dist: generate_cluster_version=20282) [2022-08-08 10:50:26.103242] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927026103233, cur_ts_=1659927026082799, delta=20434) [2022-08-08 10:50:26.142921] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:26.142970] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668266, retry interval=1000000, ret=-4179) [2022-08-08 10:50:26.145463] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:26.148998] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:26.149035] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:26.542965] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4692-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:26.543009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4692-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:26.543284] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4692-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:26.545867] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:26.545929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:26.574141] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:26.574360] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] task process failed, start retry(max retry time=9223372036854668261, retry interval=1000000, ret=-4179) [2022-08-08 10:50:27.151329] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:27.180812] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:27.182403] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:27.182455] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668265, retry interval=1000000, ret=-4179) [2022-08-08 10:50:27.183026] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:27.183041] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:27.394821] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6C-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:27.394839] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6C-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:27.394846] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:27.600053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:27.600125] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:27.601643] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:27.601705] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668260, retry interval=1000000, ret=-4179) [2022-08-08 10:50:28.185280] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:28.191084] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:28.191114] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:28.210834] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:28.210961] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:28.214107] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:28.214158] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668264, retry interval=1000000, ret=-4179) [2022-08-08 10:50:28.410816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.410845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6F-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.411241] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.411709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A6F-0-0] [lt=202] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.412865] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.413386] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.413413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.413431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:28.572073] WARN [STORAGE.TRANS] start_stmt_sanity_check_ (ob_trans_service.cpp:10302) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=99] [dc=0] transaction is timeout(trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}) [2022-08-08 10:50:28.573770] WARN [STORAGE.TRANS] start_stmt_v2 (ob_trans_service.cpp:9886) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=1645] [dc=0] sanity check fail(ret=-6210, trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}) [2022-08-08 10:50:28.573810] WARN [STORAGE.TRANS] start_stmt_v2 (ob_trans_service.cpp:9947) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=14] [dc=0] start statement failed(ret=-6210, trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}, stmt_param=[tenant_id=1001, stmt_expired_time=1659927038501834, is_retry_sql=0, safe_weak_read_snapshot=0, weak_read_snapshot_source=0, weak_read_snapshot_source_str=UNKOWN, is_trx_elr=0, multi_tenant=0]) [2022-08-08 10:50:28.573842] WARN [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:1151) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=5] [dc=0] fail start stmt(ret=-6210, trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}, pla={partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, out_pla={partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_remote=false, unreachable_partitions.count()=0) [2022-08-08 10:50:28.573856] WARN [SQL] start_stmt (ob_result_set.cpp:386) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=7] [dc=0] fail to start stmt(ret=-6210, participants={partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, physical_plan_->get_dependency_table()=[{table_id:1100611139453778, schema_version:1659926854292344, object_type:1, is_db_explicit:false, is_existed:true}]) [2022-08-08 10:50:28.573860] WARN [SQL.SESSION] do_open_plan (ob_result_set.cpp:660) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=3] [dc=0] fail start stmt(ret=-6210) [2022-08-08 10:50:28.573864] WARN [SQL.SESSION] sync_open (ob_result_set.cpp:140) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=3] [dc=0] fail to exec execute()(ret=-6210) [2022-08-08 10:50:28.573871] WARN [SERVER] test_and_save_retry_state (ob_query_retry_ctrl.cpp:527) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=1] [dc=0] do not need retry(client_ret=-6210, err=-6210, expected_stmt=true, THIS_WORKER.get_timeout_ts()=1659927038571834, retry_type_=0, result.get_stmt_type()=5, result.get_exec_context().need_change_timeout_ret()=true, session->get_retry_info().get_last_query_retry_err()=0) [2022-08-08 10:50:28.573877] WARN [SERVER] response_result (ob_sync_plan_driver.cpp:73) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=4] [dc=0] result set open failed, check if need retry(ret=-6210, cli_ret=-6210, retry_ctrl_.need_retry()=0) [2022-08-08 10:50:28.573884] WARN [SERVER] response_result (ob_sync_plan_driver.cpp:80) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=3] [dc=0] close result set fail(cret=-6210) [2022-08-08 10:50:28.573965] WARN [SERVER] do_process (obmp_query.cpp:775) [11024][0][YB42C0A80B80-0005E3CF9BC835E6-0-0] [lt=2] [dc=0] execute query fail(ret=-6210, timeout_timestamp=1659927038571834) [2022-08-08 10:50:28.622316] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10930, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10916) [2022-08-08 10:50:28.653495] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:28.662192] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668259, retry interval=1000000, ret=-4179) [2022-08-08 10:50:28.671915] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11115) [2022-08-08 10:50:28.672513] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12711, time_dist: generate_cluster_version=12708) [2022-08-08 10:50:29.195125] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:29.206011] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:29.206037] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:29.244401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:29.244601] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:29.245174] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:29.245245] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668263, retry interval=1000000, ret=-4179) [2022-08-08 10:50:29.552861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:29.553817] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:29.611578] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:50:29.612074] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=489] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:50:29.671125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:29.708572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:29.708645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:29.709874] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:29.709948] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668258, retry interval=1000000, ret=-4179) [2022-08-08 10:50:29.778000] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14602432}) [2022-08-08 10:50:29.883716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:29.885431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:29.886071] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13184) [2022-08-08 10:50:29.886255] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14270, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14259) [2022-08-08 10:50:29.933763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:29.934567] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=49999, time_dist: do_cluster_heartbeat=8, generate_cluster_version=49984, cluster_service_self_check=5) [2022-08-08 10:50:29.937091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:29.944249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:29.944795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:29.995155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:30.019270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:30.020534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:30.025430] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927028193253, end_time=1659927030025417, elapsed_time=1832164, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:50:30.051068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:30.062890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:30.083515] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:30.177294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:30.198120] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:30.207451] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:30.213032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:30.213070] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:30.272117] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:30.272242] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668262, retry interval=1000000, ret=-4179) [2022-08-08 10:50:30.388572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:30.423224] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A75-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:30.423429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A75-0-0] [lt=150] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:30.423541] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:30.423730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:30.423804] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:30.424043] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:30.440853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:30.494809] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:30.735747] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:30.735853] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] task process failed, start retry(max retry time=9223372036854668257, retry interval=1000000, ret=-4179) [2022-08-08 10:50:31.215701] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:31.219032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:31.219071] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:31.294734] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:31.294788] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668261, retry interval=1000000, ret=-4179) [2022-08-08 10:50:31.762073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:31.762277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:31.763737] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:31.763790] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668256, retry interval=1000000, ret=-4179) [2022-08-08 10:50:31.924894] WARN [STORAGE.TRANS] submit_log_impl_ (ob_trans_part_ctx.cpp:6697) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] submitting log costs too much time(ret=0, ret="OB_SUCCESS", timeguard=time guard 'submit_log_impl' cost too much time, used=31290, time_dist: 2,31240,8,3, trans_id={hash:5316279433146650829, inc:84071029, addr:"192.168.11.128:2882", t:1659927031888310}, partition={tid:1100611139404002, partition_id:0, part_cnt:0}, state={prepare_version:1659927031924808, state:0}, real_log_type=64, log_size=467) [2022-08-08 10:50:31.924914] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard 'submit_log_impl' cost too much time, used=31344, time_dist: 2,31240,8,3) [2022-08-08 10:50:31.928111] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927031928101, cur_ts_=1659927031893355, delta=34746) [2022-08-08 10:50:31.929513] TRACE [TRACE]ob_trans_ctx.cpp:487 [10630][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] [trans warn] (TRACE=begin_ts=1659927031888572 2022-08-08 02:50:31.888572|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:5316279433146650829, inc:84071029, addr:{ip:"192.168.11.128", port:2882}, t:1659927031888310}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5887-0-0, start_time:1659927031888453, snapshot:1659927031858017|[end_task] u=4972 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=12 stc:1659927031893563|[calc_checksum_by_commit] u=31242 ret:0, checksum:2591528277, checksum_log_ts:9223372036854775807|[submit_log] u=9 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=101 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=4516 trans_version:1659927031924808, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8298981, logging:0, pending:0, uref:1|[end_trans_cb] u=15 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=40899) [2022-08-08 10:50:31.931534] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43572, time_dist: generate_cluster_version=43567) [2022-08-08 10:50:32.221676] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:32.228167] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:32.228235] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:32.314354] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:32.314555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:32.315928] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:32.315989] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668260, retry interval=1000000, ret=-4179) [2022-08-08 10:50:32.434023] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:32.434093] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:32.434345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:32.620026] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:32.783387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:32.783433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:32.785262] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:32.785324] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668255, retry interval=1000000, ret=-4179) [2022-08-08 10:50:33.231032] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:33.234490] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:33.234583] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:33.251390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=955] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:33.324638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C481E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.328616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.353108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=477] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.354469] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:33.354519] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668259, retry interval=1000000, ret=-4179) [2022-08-08 10:50:33.371246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:33.381445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.410319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.564453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.573280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.573353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:33.573744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:33.586222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:33.588287] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.616268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:33.696568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.735492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.735634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:33.746013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:33.747528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:33.750845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.758793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:33.785835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:33.795613] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.795675] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.813600] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.813700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:33.814620] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:33.814674] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668254, retry interval=1000000, ret=-4179) [2022-08-08 10:50:33.818434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.839748] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:33.839783] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.840023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.842299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:33.852616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:33.852937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:33.860360] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:33.873408] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.874364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:33.874399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.876939] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:33.879847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.880221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:33.880377] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12447) [2022-08-08 10:50:33.883650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.895384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:33.929097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:33.952080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.962322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.962499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:33.983679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:34.003520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.003561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:34.006473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:34.024206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:34.035600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.129912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:34.130300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.140004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:34.202264] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:34.202389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.210638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.237474] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:34.237522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.242252] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:34.242291] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:34.290025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.307549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:34.382277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:34.382307] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:34.382413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:34.383857] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:34.383942] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] task process failed, start retry(max retry time=9223372036854668258, retry interval=1000000, ret=-4179) [2022-08-08 10:50:34.445832] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:34.445981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:34.446174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:34.446349] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:34.480383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.633921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:34.836212] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:34.836236] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:34.837571] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:34.837605] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668253, retry interval=1000000, ret=-4179) [2022-08-08 10:50:35.107100] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:50:35.245019] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:35.248211] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:35.248234] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:35.406398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:35.406482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:35.408158] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:35.408207] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668257, retry interval=1000000, ret=-4179) [2022-08-08 10:50:35.860671] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:35.860748] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:35.862127] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:35.862162] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668252, retry interval=1000000, ret=-4179) [2022-08-08 10:50:36.250920] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:36.255425] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:36.255451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:36.410162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EA8-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:36.410274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EA8-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:36.428315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:36.428560] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=229] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:36.429921] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:36.429986] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668256, retry interval=1000000, ret=-4179) [2022-08-08 10:50:36.868639] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:36.868664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:36.894892] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:36.894961] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668251, retry interval=1000000, ret=-4179) [2022-08-08 10:50:36.930497] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11406, time_dist: do_cluster_heartbeat=10, generate_cluster_version=11393) [2022-08-08 10:50:37.148204] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927037148198, cur_ts_=1659927037112856, delta=35342) [2022-08-08 10:50:37.258270] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:37.266887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:37.266912] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:37.453180] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:37.453240] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668255, retry interval=1000000, ret=-4179) [2022-08-08 10:50:37.896903] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:37.918128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:37.918478] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:37.919956] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:37.920020] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668250, retry interval=1000000, ret=-4179) [2022-08-08 10:50:38.282061] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:38.285137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:38.285179] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:38.486507] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:38.486556] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668254, retry interval=1000000, ret=-4179) [2022-08-08 10:50:38.724738] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12248, time_dist: generate_cluster_version=12243) [2022-08-08 10:50:38.965187] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:38.965241] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668249, retry interval=1000000, ret=-4179) [2022-08-08 10:50:39.122015] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19623) [2022-08-08 10:50:39.122909] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21227, time_dist: generate_cluster_version=21224) [2022-08-08 10:50:39.142167] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20107) [2022-08-08 10:50:39.142557] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31126, time_dist: do_cluster_heartbeat=11, generate_cluster_version=31109, cluster_service_self_check=4) [2022-08-08 10:50:39.290857] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:39.293941] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:39.293965] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:39.297881] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12202) [2022-08-08 10:50:39.298097] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13188, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13176) [2022-08-08 10:50:39.366301] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14748) [2022-08-08 10:50:39.366626] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15914, time_dist: generate_cluster_version=15910) [2022-08-08 10:50:39.414864] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10634) [2022-08-08 10:50:39.434283] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18193) [2022-08-08 10:50:39.512301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.512383] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.513676] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:39.513740] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668253, retry interval=1000000, ret=-4179) [2022-08-08 10:50:39.520207] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11471) [2022-08-08 10:50:39.521297] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13572, time_dist: generate_cluster_version=13568) [2022-08-08 10:50:39.539843] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19589) [2022-08-08 10:50:39.554868] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14969) [2022-08-08 10:50:39.555075] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19437, time_dist: generate_cluster_version=19435) [2022-08-08 10:50:39.575608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:39.578027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:39.627226] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:50:39.627257] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:50:39.694333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:39.748766] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14606816}) [2022-08-08 10:50:39.757794] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15845) [2022-08-08 10:50:39.758194] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17339, time_dist: generate_cluster_version=17336) [2022-08-08 10:50:39.900512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:39.905916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:39.935737] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927038194682, end_time=1659927039935723, elapsed_time=1741041, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:50:39.959118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:39.961030] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:39.962878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:39.966929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.967276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=309] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:39.967399] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.967434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.967610] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.967883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.968089] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.968193] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46EA-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.968488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.987546] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.987872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:39.993273] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:39.993432] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] task process failed, start retry(max retry time=9223372036854668248, retry interval=1000000, ret=-4179) [2022-08-08 10:50:40.013087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:40.041251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:40.041307] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:40.075366] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:40.085520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:40.106254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:40.198107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:40.217888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:40.295793] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:40.299249] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:40.299286] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:40.407402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=208] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:40.464110] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:40.512788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:40.516089] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EAE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:40.516118] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EAE-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:40.516235] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EAE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:40.516491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EAE-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:40.516517] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EAE-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:40.542173] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:40.543594] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:40.543657] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668252, retry interval=1000000, ret=-4179) [2022-08-08 10:50:41.015644] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:41.015663] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:41.017216] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:41.017266] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668247, retry interval=1000000, ret=-4179) [2022-08-08 10:50:41.303639] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:41.306799] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:41.306848] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:41.545466] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482A-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:41.545492] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482A-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:41.563787] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:41.564057] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=184] [dc=0] task process failed, start retry(max retry time=9223372036854668251, retry interval=1000000, ret=-4179) [2022-08-08 10:50:42.054438] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927042054433, cur_ts_=1659927042027074, delta=27359) [2022-08-08 10:50:42.119167] WARN [LIB] ~ObTimeGuard (utility.h:788) [10414][0][YB42C0A80B80-0005E3CFC4EA9727-0-0] [lt=14] [dc=0] destruct(*this=time guard 'rpc post' cost too much time, used=65129, time_dist: 53,61,2,65010) [2022-08-08 10:50:42.145547] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:42.145628] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668246, retry interval=1000000, ret=-4179) [2022-08-08 10:50:42.282442] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927042282432, cur_ts_=1659927042244513, delta=37919) [2022-08-08 10:50:42.323156] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:42.331058] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:42.331097] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:42.344499] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=10850) [2022-08-08 10:50:42.360554] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27615, time_dist: do_cluster_heartbeat=9, generate_cluster_version=27602) [2022-08-08 10:50:42.361919] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17077, time_dist: do_cluster_heartbeat=9, generate_cluster_version=17064) [2022-08-08 10:50:42.501025] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A99-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.501087] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A99-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.501328] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A99-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.501988] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2A99-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.502228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.502543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.506297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:42.604494] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:42.604755] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668250, retry interval=1000000, ret=-4179) [2022-08-08 10:50:43.147513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.147767] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=225] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.147807] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.148829] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.148864] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.148895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.148929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.148942] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.148953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C482D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.165684] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:43.167138] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:43.167199] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668245, retry interval=1000000, ret=-4179) [2022-08-08 10:50:43.272060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:43.350173] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:43.353976] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:43.354019] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:43.392308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:43.397803] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.426880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.585074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.606307] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.609254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:43.609264] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.609555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:43.609596] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927043609593, cur_ts_=1659927043588173, delta=21420) [2022-08-08 10:50:43.610152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:43.639784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:43.710372] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] clock out of order(cur_ts=1659927043710364, cur_ts_=1659927043669497, delta=40867) [2022-08-08 10:50:43.713663] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:43.713707] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668249, retry interval=1000000, ret=-4179) [2022-08-08 10:50:43.714964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.758870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.760086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:43.770368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:43.770574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:43.770610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.782704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:43.804490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=223] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:43.842752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.864096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=182] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:43.864361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.864399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:43.864645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.873998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=225] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:43.884304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=529] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:43.897039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:43.897061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=265] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.897780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.897798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:43.899690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.899925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.900114] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:43.918842] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927043918833, cur_ts_=1659927043898287, delta=20546) [2022-08-08 10:50:43.924603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=373] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:43.939627] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:43.947880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:43.956928] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15001) [2022-08-08 10:50:43.970214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.978627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:43.980082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:44.002459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:44.020254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.021359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:44.031640] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:44.044067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:44.051966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.152179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.152184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:44.155251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:44.188075] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:44.190093] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:44.190142] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668244, retry interval=1000000, ret=-4179) [2022-08-08 10:50:44.222865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.226601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:44.230614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.253569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.311846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.323835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:44.356580] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:44.359891] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:44.360034] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:44.389844] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20714] [dc=0] clock out of order(cur_ts=1659927044389837, cur_ts_=1659927044369388, delta=20449) [2022-08-08 10:50:44.398300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:44.497075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.649352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:44.742221] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:44.742258] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668248, retry interval=1000000, ret=-4179) [2022-08-08 10:50:45.124001] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927045123993, cur_ts_=1659927045093092, delta=30901) [2022-08-08 10:50:45.215356] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:45.215408] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668243, retry interval=1000000, ret=-4179) [2022-08-08 10:50:45.362406] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:45.366556] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:45.366584] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:45.744506] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4830-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:45.744700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4830-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:45.744717] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4830-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:45.771802] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:45.771843] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668247, retry interval=1000000, ret=-4179) [2022-08-08 10:50:46.078831] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927046078824, cur_ts_=1659927046049372, delta=29452) [2022-08-08 10:50:46.235047] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:46.236541] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:46.236688] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] task process failed, start retry(max retry time=9223372036854668242, retry interval=1000000, ret=-4179) [2022-08-08 10:50:46.368809] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:46.371985] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:46.372027] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:46.569696] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927046569690, cur_ts_=1659927046532502, delta=37188) [2022-08-08 10:50:46.775354] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46F3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:46.775433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46F3-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:46.796469] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:46.796531] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:46.797987] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:46.798026] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668246, retry interval=1000000, ret=-4179) [2022-08-08 10:50:47.259800] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.259824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.261654] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:47.261693] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668241, retry interval=1000000, ret=-4179) [2022-08-08 10:50:47.374655] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:47.377932] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:47.377971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:47.666546] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927047666541, cur_ts_=1659927047623844, delta=42697) [2022-08-08 10:50:47.669966] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46261) [2022-08-08 10:50:47.670465] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=47539, time_dist: generate_cluster_version=47536) [2022-08-08 10:50:47.802937] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46B1-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.803003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46B1-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.803025] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46B1-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.803052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46B1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.803083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46B1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:47.830689] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:47.830732] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668245, retry interval=1000000, ret=-4179) [2022-08-08 10:50:48.289803] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:48.290016] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:48.291579] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:48.291975] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668240, retry interval=1000000, ret=-4179) [2022-08-08 10:50:48.336413] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927048336404, cur_ts_=1659927048316291, delta=20113) [2022-08-08 10:50:48.382720] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:48.386105] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:48.386175] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:48.901426] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:50:48.955715] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=124788) [2022-08-08 10:50:48.955750] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] slow flush(flush_time=12, io_time=124818, file_id=237, flush_start_offset=46868720, file_offset=46869341) [2022-08-08 10:50:48.970394] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:1599644420793098088, inc:84071670, addr:"192.168.11.128:2882", t:1659927048825371}, log_id=8299065, used_time=139620) [2022-08-08 10:50:48.971602] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=142971, time_dist: generate_cluster_version=142967) [2022-08-08 10:50:48.982667] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927048982660, cur_ts_=1659927048961645, delta=21015) [2022-08-08 10:50:48.988558] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32787) [2022-08-08 10:50:48.988801] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=96542, time_dist: generate_cluster_version=96539) [2022-08-08 10:50:49.029677] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:49.029725] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668244, retry interval=1000000, ret=-4179) [2022-08-08 10:50:49.297584] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:49.312596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:49.312712] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:49.314140] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:49.314988] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=720] [dc=0] task process failed, start retry(max retry time=9223372036854668239, retry interval=1000000, ret=-4179) [2022-08-08 10:50:49.394710] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:49.398755] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:49.398803] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:49.583228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AAE-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:49.583246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AAE-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:49.583253] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AAE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:49.583259] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AAE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:49.591806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:49.605611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:49.612665] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:50:49.612700] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:50:49.710546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:49.918227] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:49.930533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:49.952451] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927049952442, cur_ts_=1659927049926134, delta=26308) [2022-08-08 10:50:49.984645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:49.991353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:49.991531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:49.991549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:50.030772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:50.058126] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46F9-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:50.058167] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C46F9-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:50.062534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:50.066410] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:50.073177] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:50.078447] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13061) [2022-08-08 10:50:50.078951] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16246, time_dist: generate_cluster_version=16243) [2022-08-08 10:50:50.084160] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20733, time_dist: generate_cluster_version=20728) [2022-08-08 10:50:50.095052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:50.105142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:50.118576] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14611104}) [2022-08-08 10:50:50.123740] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:50.123826] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:50.126305] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=477] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:50.126367] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668243, retry interval=1000000, ret=-4179) [2022-08-08 10:50:50.126221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:50.219029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:50.239545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:50.327016] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=413] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927048197215, end_time=1659927050326976, elapsed_time=2129761, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:50:50.342769] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:50.344952] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:50.345029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668238, retry interval=1000000, ret=-4179) [2022-08-08 10:50:50.400994] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:50.404130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:50.404168] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:50.425621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:50.481848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:50.534961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:50.600040] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:50.600525] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:51.063498] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14457, time_dist: generate_cluster_version=14452) [2022-08-08 10:50:51.096140] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927051096132, cur_ts_=1659927051056310, delta=39822) [2022-08-08 10:50:51.228431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:51.228524] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:51.229935] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:51.229997] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668242, retry interval=1000000, ret=-4179) [2022-08-08 10:50:51.367345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:51.367423] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:51.368920] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:51.368961] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668237, retry interval=1000000, ret=-4179) [2022-08-08 10:50:51.406867] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:51.409935] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:51.409971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:51.473693] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12295, time_dist: generate_cluster_version=12288) [2022-08-08 10:50:52.262486] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:52.262536] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668241, retry interval=1000000, ret=-4179) [2022-08-08 10:50:52.400275] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:52.400388] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] task process failed, start retry(max retry time=9223372036854668236, retry interval=1000000, ret=-4179) [2022-08-08 10:50:52.414969] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:52.418216] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:52.418252] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:52.610123] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:52.620318] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:52.892141] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927052892134, cur_ts_=1659927052870884, delta=21250) [2022-08-08 10:50:53.039497] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927053039491, cur_ts_=1659927053014612, delta=24879) [2022-08-08 10:50:53.264918] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C483C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:53.264956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C483C-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:53.287630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:53.288672] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:53.290142] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:53.290222] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854668240, retry interval=1000000, ret=-4179) [2022-08-08 10:50:53.409654] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:53.413758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.423158] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:53.443756] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:53.443821] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:53.447540] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=163] [dc=0] clock out of order(cur_ts=1659927053447532, cur_ts_=1659927053423439, delta=24093) [2022-08-08 10:50:53.449354] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.459865] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:53.463660] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:53.463736] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668235, retry interval=1000000, ret=-4179) [2022-08-08 10:50:53.465724] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12822) [2022-08-08 10:50:53.466873] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17201, time_dist: generate_cluster_version=17197) [2022-08-08 10:50:53.477475] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20752, time_dist: generate_cluster_version=20749) [2022-08-08 10:50:53.603285] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.627212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:53.627210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.627466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:53.631193] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:53.634504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.656234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:53.705926] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=12986) [2022-08-08 10:50:53.716293] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25024, time_dist: generate_cluster_version=25020) [2022-08-08 10:50:53.742841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.775987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:53.777548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.790408] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:53.790489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:53.794881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.803463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:53.824257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:53.857909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.880441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.888231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:50:53.888329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.889724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:53.891058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:53.899580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:53.916265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=137] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.916500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.917352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.917501] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:53.919443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.919731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:53.921579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:50:53.948693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=14513] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:53.979963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:53.992467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.996891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:53.997098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:54.017269] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=587] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:54.027139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:54.037255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.038430] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:54.048914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:54.066974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:50:54.071330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.071495] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12471) [2022-08-08 10:50:54.072098] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15831, time_dist: do_cluster_heartbeat=152, generate_cluster_version=15675) [2022-08-08 10:50:54.171300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.177167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:54.184278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:54.238461] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927054238457, cur_ts_=1659927054211575, delta=26882) [2022-08-08 10:50:54.239861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.247661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.249157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:54.269858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.324387] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:54.324443] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668239, retry interval=1000000, ret=-4179) [2022-08-08 10:50:54.331514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.341558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:50:54.423281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:54.448123] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:54.452188] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:54.452226] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:54.497536] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:54.497607] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668234, retry interval=1000000, ret=-4179) [2022-08-08 10:50:54.517547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.651676] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27994) [2022-08-08 10:50:54.653743] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30895, time_dist: do_cluster_heartbeat=8, generate_cluster_version=30883) [2022-08-08 10:50:54.696415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:50:54.696537] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927054696534, cur_ts_=1659927054670808, delta=25726) [2022-08-08 10:50:55.135140] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:50:55.343952] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:55.345097] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:55.345162] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668238, retry interval=1000000, ret=-4179) [2022-08-08 10:50:55.455070] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:55.458274] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:55.458317] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:55.515804] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:55.515955] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:55.517132] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:55.517193] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668233, retry interval=1000000, ret=-4179) [2022-08-08 10:50:55.664666] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AC0-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:50:55.870704] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927055870696, cur_ts_=1659927055849299, delta=21397) [2022-08-08 10:50:55.884726] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=16205) [2022-08-08 10:50:55.909229] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23770) [2022-08-08 10:50:55.909656] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=42017, time_dist: do_cluster_heartbeat=10, generate_cluster_version=42005) [2022-08-08 10:50:56.385266] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:56.385304] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668237, retry interval=1000000, ret=-4179) [2022-08-08 10:50:56.461147] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:56.464743] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=234] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:56.464868] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:56.559931] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:56.560023] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] task process failed, start retry(max retry time=9223372036854668232, retry interval=1000000, ret=-4179) [2022-08-08 10:50:57.153862] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927057153853, cur_ts_=1659927057092935, delta=60918) [2022-08-08 10:50:57.157352] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22177, time_dist: do_cluster_heartbeat=25, generate_cluster_version=22147) [2022-08-08 10:50:57.158320] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12563, time_dist: do_cluster_heartbeat=11, generate_cluster_version=12549) [2022-08-08 10:50:57.260285] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927057260260, cur_ts_=1659927057233664, delta=26596) [2022-08-08 10:50:57.419745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:57.419825] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:57.420953] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:57.421019] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668236, retry interval=1000000, ret=-4179) [2022-08-08 10:50:57.467298] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:57.470932] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:57.470955] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:57.574714] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:57.594365] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:57.594438] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668231, retry interval=1000000, ret=-4179) [2022-08-08 10:50:58.186150] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] clock out of order(cur_ts=1659927058186145, cur_ts_=1659927058155750, delta=30395) [2022-08-08 10:50:58.432509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:58.432575] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:58.451909] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:58.453284] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:58.453333] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668235, retry interval=1000000, ret=-4179) [2022-08-08 10:50:58.474836] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:58.478414] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:58.478664] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=238] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:58.547266] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10451, time_dist: do_cluster_heartbeat=11, generate_cluster_version=10436) [2022-08-08 10:50:58.643231] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927058643223, cur_ts_=1659927058594162, delta=49061) [2022-08-08 10:50:58.660496] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:58.660759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=247] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:58.664330] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:58.664394] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668230, retry interval=1000000, ret=-4179) [2022-08-08 10:50:59.375135] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31987, time_dist: generate_cluster_version=31982) [2022-08-08 10:50:59.375647] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] clock out of order(cur_ts=1659927059375642, cur_ts_=1659927059343685, delta=31957) [2022-08-08 10:50:59.479363] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:59.479515] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:50:59.479956] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:59.480014] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668234, retry interval=1000000, ret=-4179) [2022-08-08 10:50:59.481642] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:50:59.485854] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:50:59.485898] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:50:59.520854] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927059520848, cur_ts_=1659927059493437, delta=27411) [2022-08-08 10:50:59.616927] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=275] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:59.620744] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15546) [2022-08-08 10:50:59.622047] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18104, time_dist: generate_cluster_version=18101) [2022-08-08 10:50:59.625583] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:50:59.625628] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:50:59.646785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:59.725378] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927059725360, cur_ts_=1659927059704383, delta=20977) [2022-08-08 10:50:59.751013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:50:59.764335] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=141] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17734) [2022-08-08 10:50:59.764713] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=39066, time_dist: do_cluster_heartbeat=9, generate_cluster_version=39054) [2022-08-08 10:50:59.786771] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22269) [2022-08-08 10:50:59.792529] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43819, time_dist: do_cluster_heartbeat=11, generate_cluster_version=43803) [2022-08-08 10:50:59.820759] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:50:59.820813] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668229, retry interval=1000000, ret=-4179) [2022-08-08 10:50:59.861255] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=68214) [2022-08-08 10:50:59.862593] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927059862582, cur_ts_=1659927059820977, delta=41605) [2022-08-08 10:50:59.877664] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16358) [2022-08-08 10:50:59.935594] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:50:59.945797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:00.002497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.006989] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:00.015109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.015136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:00.046007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:00.085353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.085378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:00.110697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.111115] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927060111111, cur_ts_=1659927060089917, delta=21194) [2022-08-08 10:51:00.121359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.137605] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27128) [2022-08-08 10:51:00.138946] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=54631, time_dist: do_cluster_heartbeat=10, generate_cluster_version=54617) [2022-08-08 10:51:00.139181] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26803, time_dist: do_cluster_heartbeat=38, generate_cluster_version=26763) [2022-08-08 10:51:00.151480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=1146] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:00.227901] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14615520}) [2022-08-08 10:51:00.256965] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.257103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:00.419797] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927058201969, end_time=1659927060419783, elapsed_time=2217814, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:51:00.443991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:00.485654] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:00.487447] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:00.492894] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:00.492934] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:00.497950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:00.510421] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=493] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:00.512059] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:00.512095] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668233, retry interval=1000000, ret=-4179) [2022-08-08 10:51:00.552889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:00.767791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ACF-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.767824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ACF-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.767962] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ACF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.768091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ACF-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.768189] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.768401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.768461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.768549] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:00.822895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4708-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:00.842110] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:00.842167] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668228, retry interval=1000000, ret=-4179) [2022-08-08 10:51:01.495325] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:01.498563] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:01.498666] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:01.543675] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:01.543734] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668232, retry interval=1000000, ret=-4179) [2022-08-08 10:51:01.869227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:01.869401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:01.872837] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:01.872875] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668227, retry interval=1000000, ret=-4179) [2022-08-08 10:51:02.501036] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:02.504951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:02.505001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:02.545772] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C470B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.545926] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C470B-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.546479] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C470B-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.564568] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.564611] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.565890] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:02.565941] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668231, retry interval=1000000, ret=-4179) [2022-08-08 10:51:02.789266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD5-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:02.789300] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD5-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:02.789399] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:02.789526] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD5-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:02.789837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:02.875396] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46C6-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.875575] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46C6-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.896544] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.896587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:02.898393] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:02.898429] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668226, retry interval=1000000, ret=-4179) [2022-08-08 10:51:03.304907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:03.432090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:03.451662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.473507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.511986] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:03.524482] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:03.524528] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:03.600709] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927063600682, cur_ts_=1659927063578194, delta=22488) [2022-08-08 10:51:03.603415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:03.603439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:03.636018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.646553] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:03.646594] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668230, retry interval=1000000, ret=-4179) [2022-08-08 10:51:03.647168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.647185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:03.647389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:03.649051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:03.657485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.680291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:03.766411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.794945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.796053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:03.797010] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD8-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.797088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD8-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.797315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD8-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.797483] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AD8-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.797644] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.797816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.798046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.798188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:03.807595] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:03.815561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:03.817091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.824954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:03.847293] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:03.878344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.899077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.908266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:03.909570] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:03.909616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:03.915908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:03.918411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:03.918792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:03.918819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.929165] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10385) [2022-08-08 10:51:03.929931] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12039, time_dist: generate_cluster_version=12037) [2022-08-08 10:51:03.931793] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:03.931889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:03.933188] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:03.933248] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668225, retry interval=1000000, ret=-4179) [2022-08-08 10:51:03.937601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.937674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:03.937707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.938596] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:03.939591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:03.939694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:03.941675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:03.968893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:04.002405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:04.012496] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:04.012575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:04.023013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:04.051613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:04.053953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:04.053995] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.070975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:04.081222] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:04.084952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:04.095510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.188575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.192842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:04.208911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:04.255772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.268926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:04.269151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.288752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=227] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.350758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=527] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.362095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:04.440531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:04.526853] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:04.530155] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:04.530287] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:04.536743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.649864] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46C9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:04.649900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46C9-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:04.668201] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:04.670001] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:04.670055] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668229, retry interval=1000000, ret=-4179) [2022-08-08 10:51:04.712422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:04.807775] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24460) [2022-08-08 10:51:04.808269] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27647, time_dist: generate_cluster_version=27644) [2022-08-08 10:51:04.808283] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927064808278, cur_ts_=1659927064784692, delta=23586) [2022-08-08 10:51:04.970643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:04.972484] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:04.973481] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=321] [dc=0] task process failed, start retry(max retry time=9223372036854668224, retry interval=1000000, ret=-4179) [2022-08-08 10:51:05.532714] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:05.535724] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:05.535759] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:05.672769] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C484E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:05.673207] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C484E-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:05.673257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C484E-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:05.673744] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C484E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:05.693709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=193] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:05.696467] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:05.696514] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668228, retry interval=1000000, ret=-4179) [2022-08-08 10:51:05.998188] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:05.998230] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668223, retry interval=1000000, ret=-4179) [2022-08-08 10:51:06.191278] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13517) [2022-08-08 10:51:06.192436] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15476, time_dist: generate_cluster_version=15472) [2022-08-08 10:51:06.540166] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:06.543236] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:06.543280] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:06.718330] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=1467] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:06.718438] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:06.719865] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:06.719926] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668227, retry interval=1000000, ret=-4179) [2022-08-08 10:51:07.003337] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:07.006052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:07.026033] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:07.026204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:07.027789] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:07.027849] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668222, retry interval=1000000, ret=-4179) [2022-08-08 10:51:07.313119] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659927067313111, cur_ts_=1659927067280148, delta=32963) [2022-08-08 10:51:07.545536] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:07.548744] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:07.548791] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:07.745250] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:07.745299] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:07.746557] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:07.746665] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] task process failed, start retry(max retry time=9223372036854668226, retry interval=1000000, ret=-4179) [2022-08-08 10:51:07.860854] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.860945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=76] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.860955] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.861027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.861588] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.861635] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.861642] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.864296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AE4-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.865942] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=280] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.866301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:07.866374] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:08.031461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4851-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:08.031498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4851-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:08.031520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4851-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:08.031538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4851-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:08.054814] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:08.056287] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:08.056324] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668221, retry interval=1000000, ret=-4179) [2022-08-08 10:51:08.187466] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13920) [2022-08-08 10:51:08.196690] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24364, time_dist: generate_cluster_version=24359) [2022-08-08 10:51:08.288847] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927068288839, cur_ts_=1659927068266697, delta=22142) [2022-08-08 10:51:08.390415] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=159336) [2022-08-08 10:51:08.409718] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:14300153560453615466, inc:84072411, addr:"192.168.11.128:2882", t:1659927068230244}, log_id=8299163, used_time=178883) [2022-08-08 10:51:08.411072] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=19235] [dc=0] slow flush(flush_time=643, io_time=178582, file_id=237, flush_start_offset=47053499, file_offset=47054121) [2022-08-08 10:51:08.414570] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=186333, time_dist: do_cluster_heartbeat=147, generate_cluster_version=186182) [2022-08-08 10:51:08.456028] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43775) [2022-08-08 10:51:08.459201] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=97060, time_dist: generate_cluster_version=96959) [2022-08-08 10:51:08.553185] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:08.556536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:08.556713] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=162] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:08.815030] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=166] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13090, time_dist: generate_cluster_version=13074) [2022-08-08 10:51:08.816621] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:08.816662] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668225, retry interval=1000000, ret=-4179) [2022-08-08 10:51:08.895326] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:08.895545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:09.077579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:09.077621] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:09.078839] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:09.078945] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668220, retry interval=1000000, ret=-4179) [2022-08-08 10:51:09.560260] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:09.577531] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:09.578510] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:09.612424] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:51:09.612443] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:51:09.636617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:09.670522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:09.769976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:09.837491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:09.837614] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:09.842001] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:09.842038] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668224, retry interval=1000000, ret=-4179) [2022-08-08 10:51:09.958554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:09.972469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:09.987002] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927069986994, cur_ts_=1659927069949795, delta=37199) [2022-08-08 10:51:10.021888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.037976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:10.038374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.055812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:10.070069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:10.116584] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.116935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:10.117509] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927070117505, cur_ts_=1659927070075906, delta=41599) [2022-08-08 10:51:10.128846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.158768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.172397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=259] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:10.174788] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:10.174827] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668219, retry interval=1000000, ret=-4179) [2022-08-08 10:51:10.215826] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43779) [2022-08-08 10:51:10.217200] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=46712, time_dist: do_cluster_heartbeat=5, generate_cluster_version=46702) [2022-08-08 10:51:10.218828] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] clock out of order(cur_ts=1659927070218818, cur_ts_=1659927070191584, delta=27234) [2022-08-08 10:51:10.226810] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10939) [2022-08-08 10:51:10.229483] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=59735, time_dist: do_cluster_heartbeat=11, generate_cluster_version=59719) [2022-08-08 10:51:10.273966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:10.281385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.459362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:10.518234] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14619968}) [2022-08-08 10:51:10.526870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:10.571276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:10.582979] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:10.587595] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=589] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22097) [2022-08-08 10:51:10.588147] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23631, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23617) [2022-08-08 10:51:10.590081] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=418] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:10.590122] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:10.728979] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=251] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927068202905, end_time=1659927070728966, elapsed_time=2526061, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:51:10.870091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:10.870191] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:10.871540] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:10.871610] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] task process failed, start retry(max retry time=9223372036854668223, retry interval=1000000, ret=-4179) [2022-08-08 10:51:11.180539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46D2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:11.183071] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:11.183168] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:11.195403] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AED-0-0] [lt=2015] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:11.203960] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:11.204036] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] task process failed, start retry(max retry time=9223372036854668218, retry interval=1000000, ret=-4179) [2022-08-08 10:51:11.406002] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] [trans sampling] (TRACE=begin_ts=1659927071401958 2022-08-08 02:51:11.401958|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:10805054783369304382, inc:84072519, addr:{ip:"192.168.11.128", port:2882}, t:1659927071401000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A594C-0-0, start_time:1659927071401657, snapshot:1659927071321508|[end_task] u=149 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927071402127|[calc_checksum_by_commit] u=25 ret:0, checksum:2380608041, checksum_log_ts:9223372036854775807|[submit_log] u=73 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3705 trans_version:1659927071402154, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8299178, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=4001) [2022-08-08 10:51:11.598935] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:11.602944] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=601] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:11.602987] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:11.895529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:11.895580] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:11.897119] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:11.897222] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854668222, retry interval=1000000, ret=-4179) [2022-08-08 10:51:12.266894] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:12.267014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:12.268401] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:12.268460] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668217, retry interval=1000000, ret=-4179) [2022-08-08 10:51:12.606252] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:12.609321] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:12.609375] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:12.620612] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:12.921653] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:12.921726] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:12.923484] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:12.923535] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668221, retry interval=1000000, ret=-4179) [2022-08-08 10:51:13.153022] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927073153013, cur_ts_=1659927073124471, delta=28542) [2022-08-08 10:51:13.260038] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.260075] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.260502] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.260519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.260538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.260547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.260552] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.262305] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.262387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.262759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF3-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:13.271535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C471A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:13.271671] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C471A-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:13.290172] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:13.290211] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668216, retry interval=1000000, ret=-4179) [2022-08-08 10:51:13.322411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:13.456811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:13.470503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.494997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.612172] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:13.616148] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:13.616176] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:13.658237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.669127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.672550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:13.672591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:13.673577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:13.673593] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.699502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:13.770485] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927073770475, cur_ts_=1659927073732797, delta=37678) [2022-08-08 10:51:13.790953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.814645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:13.816054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.832335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:13.832333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.839280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:13.845980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=157] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:13.863589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:13.903273] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927073903263, cur_ts_=1659927073876958, delta=26305) [2022-08-08 10:51:13.903717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.914535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.929386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:13.939053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.939540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:13.939601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:13.940228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:13.961797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:13.961834] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:13.962044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.962091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:13.963985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:13.964878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.965249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.987229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:13.987693] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:13.987729] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668220, retry interval=1000000, ret=-4179) [2022-08-08 10:51:14.019028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:14.032842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:14.035807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:14.039453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=458] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:14.040199] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13987, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13967, cluster_service_self_check=6) [2022-08-08 10:51:14.067061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:14.073578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:14.073952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.090554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:14.100976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:14.115179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.160414] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:14.208121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.217728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:14.229286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:14.279274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.291338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.291977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:14.310313] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:14.310415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:14.311888] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:14.311926] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668215, retry interval=1000000, ret=-4179) [2022-08-08 10:51:14.312969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.372589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.378101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:14.423010] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] [trans sampling] (TRACE=begin_ts=1659927074419687 2022-08-08 02:51:14.419687|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:12003294617785919918, inc:84072640, addr:{ip:"192.168.11.128", port:2882}, t:1659927074418000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A595B-0-0, start_time:1659927074419568, snapshot:1659927074379604|[end_task] u=123 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927074419828|[calc_checksum_by_commit] u=21 ret:0, checksum:3692962928, checksum_log_ts:9223372036854775807|[submit_log] u=56 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3009 trans_version:1659927074419850, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8299193, logging:0, pending:0, uref:1|[end_trans_cb] u=22 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=3271) [2022-08-08 10:51:14.463168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:14.560487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:14.618954] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:14.622539] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:14.622599] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:14.730251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:15.016510] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927075016506, cur_ts_=1659927074980712, delta=35794) [2022-08-08 10:51:15.034359] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:15.034493] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:15.038566] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:15.038607] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668219, retry interval=1000000, ret=-4179) [2022-08-08 10:51:15.159397] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:51:15.270874] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12976, time_dist: generate_cluster_version=12972) [2022-08-08 10:51:15.285364] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF9-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.285395] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF9-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.285642] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.285703] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2AF9-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.285795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.286593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.286623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:15.344776] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:15.344854] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668214, retry interval=1000000, ret=-4179) [2022-08-08 10:51:15.626855] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:15.632346] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:15.632384] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:15.876620] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927075876614, cur_ts_=1659927075855071, delta=21543) [2022-08-08 10:51:16.044453] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.044491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.063019] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.064866] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:16.064906] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668218, retry interval=1000000, ret=-4179) [2022-08-08 10:51:16.294686] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927076293009 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927076292999 abs_expired_time=1659927078223947 stmt_timeout=1930948 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21586783 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927076291728 start_version=-1 trans_version=1659927076292999 commit_version=0 stmt_start_time=1659927076292999 abs_expired_time=1659927106193763 stmt_timeout=29900764 abs_lock_wait_timeout=1659927106193763 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:14652455566349555243, inc:84072703, addr:"192.168.11.128:2882", t:1659927076292999} checksum_log_ts=9223372036854775807, version=1659927076293009) [2022-08-08 10:51:16.348380] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C485D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.349055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C485D-0-0] [lt=647] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.349064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C485D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.366699] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.366926] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:16.368325] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:16.368375] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668213, retry interval=1000000, ret=-4179) [2022-08-08 10:51:16.635217] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:16.638813] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:16.638850] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:17.067132] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46DB-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.067344] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46DB-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.067512] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46DB-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.067745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46DB-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.068507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46DB-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.068535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46DB-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.088062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.089787] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:17.089826] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668217, retry interval=1000000, ret=-4179) [2022-08-08 10:51:17.158785] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] clock out of order(cur_ts=1659927077158780, cur_ts_=1659927077127282, delta=31498) [2022-08-08 10:51:17.399188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.399241] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:17.400638] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:17.400700] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668212, retry interval=1000000, ret=-4179) [2022-08-08 10:51:17.607441] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927077607433, cur_ts_=1659927077570115, delta=37318) [2022-08-08 10:51:17.641580] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:17.645566] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:17.645594] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:17.841132] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927077841121, cur_ts_=1659927077801776, delta=39345) [2022-08-08 10:51:17.904622] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11142, time_dist: do_cluster_heartbeat=94, generate_cluster_version=11044) [2022-08-08 10:51:18.002871] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927078002861, cur_ts_=1659927077929389, delta=73472) [2022-08-08 10:51:18.084040] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11647) [2022-08-08 10:51:18.084276] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19565, time_dist: print_stat=5, do_cluster_heartbeat=2, generate_cluster_version=19557) [2022-08-08 10:51:18.147383] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26349) [2022-08-08 10:51:18.148473] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28149, time_dist: do_cluster_heartbeat=9, generate_cluster_version=28131, cluster_service_self_check=5) [2022-08-08 10:51:18.200750] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] clock out of order(cur_ts=1659927078200746, cur_ts_=1659927078179838, delta=20908) [2022-08-08 10:51:18.273979] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:18.274026] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668216, retry interval=1000000, ret=-4179) [2022-08-08 10:51:18.302709] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927078302693, cur_ts_=1659927078231828, delta=70865) [2022-08-08 10:51:18.334519] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=57013) [2022-08-08 10:51:18.356339] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21774) [2022-08-08 10:51:18.356600] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30676, time_dist: do_cluster_heartbeat=10, generate_cluster_version=30664) [2022-08-08 10:51:18.356648] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29242, time_dist: generate_cluster_version=29241) [2022-08-08 10:51:18.396482] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23022) [2022-08-08 10:51:18.461875] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927078461869, cur_ts_=1659927078438660, delta=23209) [2022-08-08 10:51:18.485026] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=70011) [2022-08-08 10:51:18.485108] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:18.485139] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668211, retry interval=1000000, ret=-4179) [2022-08-08 10:51:18.524975] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39911) [2022-08-08 10:51:18.530292] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=46770, time_dist: do_cluster_heartbeat=9, generate_cluster_version=46757) [2022-08-08 10:51:18.533432] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=49139, time_dist: generate_cluster_version=49135) [2022-08-08 10:51:18.535607] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10592) [2022-08-08 10:51:18.564653] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927078564647, cur_ts_=1659927078533250, delta=31397) [2022-08-08 10:51:18.651822] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:18.688728] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927078688717, cur_ts_=1659927078651504, delta=37213) [2022-08-08 10:51:18.690547] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=45212) [2022-08-08 10:51:18.695763] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=53629, time_dist: generate_cluster_version=53624) [2022-08-08 10:51:18.704378] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=257] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:18.704415] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:18.781071] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=69790) [2022-08-08 10:51:18.782636] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=73446, time_dist: do_cluster_heartbeat=8, generate_cluster_version=73434) [2022-08-08 10:51:18.796015] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=39646, time_dist: do_cluster_heartbeat=10, generate_cluster_version=39633) [2022-08-08 10:51:18.821832] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927078821824, cur_ts_=1659927078795537, delta=26287) [2022-08-08 10:51:18.935300] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927078935294, cur_ts_=1659927078905279, delta=30015) [2022-08-08 10:51:18.945095] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=44633) [2022-08-08 10:51:18.954106] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=64661, time_dist: do_cluster_heartbeat=10, generate_cluster_version=64647) [2022-08-08 10:51:18.968322] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23136) [2022-08-08 10:51:18.968913] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=50627, time_dist: generate_cluster_version=50625) [2022-08-08 10:51:19.066948] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927079066940, cur_ts_=1659927079038494, delta=28446) [2022-08-08 10:51:19.092446] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14143, time_dist: do_cluster_heartbeat=9, generate_cluster_version=14129) [2022-08-08 10:51:19.105552] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14953, time_dist: print_stat=5, do_cluster_heartbeat=3, generate_cluster_version=14941) [2022-08-08 10:51:19.194927] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2173] [dc=0] clock out of order(cur_ts=1659927079194916, cur_ts_=1659927079118829, delta=76087) [2022-08-08 10:51:19.195038] WARN [LIB] ~ObTimeGuard (utility.h:788) [10868][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] destruct(*this=time guard 'ObRpcHandler::process' cost too much time, used=47569, time_dist: 0,0,0,47566,0) [2022-08-08 10:51:19.195050] WARN [RPC.FRAME] process (ob_req_handler.cpp:209) [10868][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] process handler cost too much time(diff=47615) [2022-08-08 10:51:19.195056] WARN __tg_cleanup (easy_connection.c:3247) [10868][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] easy cost too much time: 47550us server_process [2022-08-08 10:51:19.195062] WARN __tg_cleanup (easy_connection.c:1542) [10868][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] easy cost too much time: 47566us on_readable: 0x7f686b3e0b70 [2022-08-08 10:51:19.195065] WARN ev_invoke_pending (ev.c:2137) [10868][0][Y0-0000000000000000-0-0] [lt=1] [dc=0] epoll handler too much time :47641 count=1 malloc=1/6 write=0/0 read=2/1 server_process=47550/1 client_cb=0/0 [2022-08-08 10:51:19.319988] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927079319983, cur_ts_=1659927079232565, delta=87418) [2022-08-08 10:51:19.583957] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] clock out of order(cur_ts=1659927079583949, cur_ts_=1659927079405862, delta=178087) [2022-08-08 10:51:19.661031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:19.691701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:19.747766] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927079747758, cur_ts_=1659927079665272, delta=82486) [2022-08-08 10:51:19.760019] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:19.816832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:19.825149] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=282153) [2022-08-08 10:51:19.825181] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=11, io_time=282176, file_id=237, flush_start_offset=47151869, file_offset=47152490) [2022-08-08 10:51:19.846633] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:6521117315222818045, inc:84072829, addr:"192.168.11.128:2882", t:1659927079405862}, log_id=8299217, used_time=304438) [2022-08-08 10:51:19.846813] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:19.846869] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668215, retry interval=1000000, ret=-4179) [2022-08-08 10:51:19.850256] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927079850222, cur_ts_=1659927079784484, delta=65738) [2022-08-08 10:51:19.859836] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=318204, time_dist: do_cluster_heartbeat=6, generate_cluster_version=318196) [2022-08-08 10:51:19.871980] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46784) [2022-08-08 10:51:19.872092] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=13] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:16658073285173796052, inc:84072830, addr:"192.168.11.128:2882", t:1659927079405862}, log_id=8313808, used_time=329245) [2022-08-08 10:51:19.892617] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:51:19.892640] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:51:19.892785] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=350531, time_dist: do_cluster_heartbeat=6, generate_cluster_version=350523) [2022-08-08 10:51:19.869421] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:19.905636] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36199] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:19.926130] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=54112) [2022-08-08 10:51:19.974848] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927079974838, cur_ts_=1659927079947487, delta=27351) [2022-08-08 10:51:19.975337] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:20.017901] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=91655) [2022-08-08 10:51:20.018165] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12111] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=94730, time_dist: do_cluster_heartbeat=8, generate_cluster_version=94720) [2022-08-08 10:51:20.018609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:20.039327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.043337] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:20.043377] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668210, retry interval=1000000, ret=-4179) [2022-08-08 10:51:20.069226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:20.069693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.069884] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=48838) [2022-08-08 10:51:20.073491] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=102991, time_dist: do_cluster_heartbeat=9, generate_cluster_version=102977) [2022-08-08 10:51:20.076296] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] clock out of order(cur_ts=1659927080076289, cur_ts_=1659927080045873, delta=30416) [2022-08-08 10:51:20.079306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:20.095161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:20.128333] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=58382) [2022-08-08 10:51:20.134052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.134557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:20.147665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.151967] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=439] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23155) [2022-08-08 10:51:20.152200] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=78300, time_dist: do_cluster_heartbeat=6, generate_cluster_version=78292) [2022-08-08 10:51:20.190768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:20.191205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.192598] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927080192589, cur_ts_=1659927080170431, delta=22158) [2022-08-08 10:51:20.198219] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46210) [2022-08-08 10:51:20.212583] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=79166, time_dist: do_cluster_heartbeat=14, generate_cluster_version=79150) [2022-08-08 10:51:20.289529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:20.375295] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=687] [dc=0] clock out of order(cur_ts=1659927080375285, cur_ts_=1659927080329901, delta=45384) [2022-08-08 10:51:20.375485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.439676] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=162388) [2022-08-08 10:51:20.439711] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=15, io_time=162416, file_id=237, flush_start_offset=47156957, file_offset=47157577) [2022-08-08 10:51:20.439796] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:9169219441886499780, inc:84073058, addr:"192.168.11.128:2882", t:1659927080276082}, log_id=8299220, used_time=162934) [2022-08-08 10:51:20.450245] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=174250, time_dist: do_cluster_heartbeat=13, generate_cluster_version=174232) [2022-08-08 10:51:20.501148] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927080501141, cur_ts_=1659927080436700, delta=64441) [2022-08-08 10:51:20.501250] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=61523) [2022-08-08 10:51:20.501301] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:12795177842634517812, inc:84073059, addr:"192.168.11.128:2882", t:1659927080279458}, log_id=8313811, used_time=204304) [2022-08-08 10:51:20.501513] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=224467, time_dist: do_cluster_heartbeat=9, generate_cluster_version=224455) [2022-08-08 10:51:20.501563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:20.554334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:20.605108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:20.613373] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28940) [2022-08-08 10:51:20.616426] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927080616416, cur_ts_=1659927080586204, delta=30212) [2022-08-08 10:51:20.623684] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=154] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=40064, time_dist: do_cluster_heartbeat=11, generate_cluster_version=40049) [2022-08-08 10:51:20.651003] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37574) [2022-08-08 10:51:20.656820] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=51576, time_dist: do_cluster_heartbeat=11, generate_cluster_version=51561) [2022-08-08 10:51:20.735948] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927080735942, cur_ts_=1659927080650435, delta=85507) [2022-08-08 10:51:20.840525] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=99386) [2022-08-08 10:51:20.840689] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=7] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:16435509786212714775, inc:84073086, addr:"192.168.11.128:2882", t:1659927080650435}, log_id=8313813, used_time=110748) [2022-08-08 10:51:20.840880] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=113117, time_dist: do_cluster_heartbeat=8, generate_cluster_version=113107) [2022-08-08 10:51:20.876380] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927080876374, cur_ts_=1659927080735942, delta=140432) [2022-08-08 10:51:20.903749] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=63131) [2022-08-08 10:51:20.903846] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:9982899048060320983, inc:84073096, addr:"192.168.11.128:2882", t:1659927080650435}, log_id=8299222, used_time=161729) [2022-08-08 10:51:20.903959] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=162665, time_dist: do_cluster_heartbeat=11, generate_cluster_version=162653) [2022-08-08 10:51:21.038965] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=135174) [2022-08-08 10:51:21.039001] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=14, io_time=135202, file_id=237, flush_start_offset=47160672, file_offset=47162770) [2022-08-08 10:51:21.039119] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=7] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:13931564759658415839, inc:84073111, addr:"192.168.11.128:2882", t:1659927080735942}, log_id=1655298, used_time=146528) [2022-08-08 10:51:21.039565] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:21.040026] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927081040022, cur_ts_=1659927080954883, delta=85139) [2022-08-08 10:51:21.110990] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927080992296 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927080954883 abs_expired_time=1659927083039955 stmt_timeout=2085072 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21586839 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927080992255 start_version=-1 trans_version=1659927080992256 commit_version=0 stmt_start_time=1659927080954883 abs_expired_time=1659927110940405 stmt_timeout=29985522 abs_lock_wait_timeout=1659927110940405 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:9912348600765819028, inc:84073183, addr:"192.168.11.128:2882", t:1659927080954883} checksum_log_ts=9223372036854775807, version=1659927080992296) [2022-08-08 10:51:21.136764] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=97747) [2022-08-08 10:51:21.137023] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=6] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:7366952520391127724, inc:84073152, addr:"192.168.11.128:2882", t:1659927080903953}, log_id=8313814, used_time=190348) [2022-08-08 10:51:21.137104] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=10] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:4701707356798191370, inc:84073168, addr:"192.168.11.128:2882", t:1659927080903953}, log_id=8299223, used_time=169790) [2022-08-08 10:51:21.137209] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=193158, time_dist: do_cluster_heartbeat=119, generate_cluster_version=193038) [2022-08-08 10:51:21.137259] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=170498, time_dist: print_stat=5, do_cluster_heartbeat=3, generate_cluster_version=170488, cluster_service_self_check=2) [2022-08-08 10:51:21.139548] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:21.139573] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:21.141936] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=74452] [dc=0] clock out of order(cur_ts=1659927081141855, cur_ts_=1659927080954883, delta=186972) [2022-08-08 10:51:21.171820] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35014) [2022-08-08 10:51:21.171886] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:9912348600765819028, inc:84073183, addr:"192.168.11.128:2882", t:1659927080954883}, log_id=1655299, used_time=130941) [2022-08-08 10:51:21.248982] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927081248977, cur_ts_=1659927081167369, delta=81608) [2022-08-08 10:51:21.259327] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=3148] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:21.263922] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4515] [dc=0] task process failed, start retry(max retry time=9223372036854668214, retry interval=1000000, ret=-4179) [2022-08-08 10:51:21.285898] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:21.285931] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668209, retry interval=1000000, ret=-4179) [2022-08-08 10:51:21.293048] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32941) [2022-08-08 10:51:21.352238] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927081352228, cur_ts_=1659927081299818, delta=52410) [2022-08-08 10:51:21.382094] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=89005) [2022-08-08 10:51:21.396239] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=14095] [dc=0] slow flush(flush_time=28, io_time=103110, file_id=237, flush_start_offset=47165436, file_offset=47166627) [2022-08-08 10:51:21.396788] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:16214667437316268991, inc:84073227, addr:"192.168.11.128:2882", t:1659927081248977}, log_id=8299224, used_time=136177) [2022-08-08 10:51:21.397140] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:11337497908701078962, inc:84073228, addr:"192.168.11.128:2882", t:1659927081261388}, log_id=8313815, used_time=135438) [2022-08-08 10:51:21.409793] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=149935, time_dist: do_cluster_heartbeat=72, generate_cluster_version=149860) [2022-08-08 10:51:21.414600] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=154020, time_dist: do_cluster_heartbeat=6, generate_cluster_version=154009) [2022-08-08 10:51:21.439105] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42519) [2022-08-08 10:51:21.439239] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=4] [dc=0] transaction log sync use too much time(partition={tid:1099511627916, partition_id:0, part_cnt:0}, trans_id={hash:13952815308597870809, inc:84073251, addr:"192.168.11.128:2882", t:1659927081268102}, log_id=987100, used_time=144934) [2022-08-08 10:51:21.498071] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927081498059, cur_ts_=1659927081459039, delta=39020) [2022-08-08 10:51:21.599107] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927081599096, cur_ts_=1659927081554323, delta=44773) [2022-08-08 10:51:21.670991] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=111113) [2022-08-08 10:51:21.671025] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] slow flush(flush_time=13, io_time=111140, file_id=237, flush_start_offset=47167315, file_offset=47167931) [2022-08-08 10:51:21.688836] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:9233755584690968336, inc:84073355, addr:"192.168.11.128:2882", t:1659927081459039}, log_id=8313816, used_time=148484) [2022-08-08 10:51:21.705193] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14624128}) [2022-08-08 10:51:21.745083] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] clock out of order(cur_ts=1659927081745075, cur_ts_=1659927081703634, delta=41441) [2022-08-08 10:51:21.801595] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=130549) [2022-08-08 10:51:21.801626] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=11, io_time=130574, file_id=237, flush_start_offset=47167931, file_offset=47168553) [2022-08-08 10:51:21.801684] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:14930481331766647219, inc:84073356, addr:"192.168.11.128:2882", t:1659927081459039}, log_id=8299225, used_time=247729) [2022-08-08 10:51:21.801863] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=262345, time_dist: do_cluster_heartbeat=10, generate_cluster_version=262334) [2022-08-08 10:51:21.871386] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=318144, time_dist: do_cluster_heartbeat=12, generate_cluster_version=318126) [2022-08-08 10:51:21.872300] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927081872292, cur_ts_=1659927081801562, delta=70730) [2022-08-08 10:51:21.923134] WARN [LIB] ~ObTimeGuard (utility.h:788) [10551][0][YB42C0A80B80-0005E3CF97595355-0-0] [lt=12] [dc=0] destruct(*this=time guard 'rpc post' cost too much time, used=20694, time_dist: 8,1,1,20678) [2022-08-08 10:51:21.984325] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927081984293, cur_ts_=1659927081941358, delta=42935) [2022-08-08 10:51:21.994571] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=122176) [2022-08-08 10:51:21.994609] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] slow flush(flush_time=13, io_time=122206, file_id=237, flush_start_offset=47168553, file_offset=47169169) [2022-08-08 10:51:21.994877] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:2830453568882750891, inc:84073362, addr:"192.168.11.128:2882", t:1659927081801562}, log_id=8313817, used_time=130545) [2022-08-08 10:51:21.995080] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=137914, time_dist: do_cluster_heartbeat=21, generate_cluster_version=137892) [2022-08-08 10:51:22.062371] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=67740) [2022-08-08 10:51:22.067514] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:16365629236271462364, inc:84073364, addr:"192.168.11.128:2882", t:1659927081934998}, log_id=8299226, used_time=130681) [2022-08-08 10:51:22.075199] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=139991, time_dist: do_cluster_heartbeat=27, generate_cluster_version=139960) [2022-08-08 10:51:22.076086] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21675, time_dist: do_cluster_heartbeat=10, generate_cluster_version=21662) [2022-08-08 10:51:22.167782] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16719, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16704) [2022-08-08 10:51:22.199064] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927078236377, end_time=1659927082199055, elapsed_time=3962678, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:51:22.207075] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:22.211245] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:22.211268] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:22.293018] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:22.293083] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668213, retry interval=1000000, ret=-4179) [2022-08-08 10:51:22.311060] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:22.311127] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668208, retry interval=1000000, ret=-4179) [2022-08-08 10:51:22.321626] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27202) [2022-08-08 10:51:22.355650] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13251) [2022-08-08 10:51:22.355833] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17805, time_dist: do_cluster_heartbeat=8, generate_cluster_version=17795) [2022-08-08 10:51:23.225612] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:23.228935] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:23.228980] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:23.296837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4726-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.296938] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4726-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.299661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.299707] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.313377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46E4-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.313476] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46E4-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.324458] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.324580] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.325970] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:23.326042] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668212, retry interval=1000000, ret=-4179) [2022-08-08 10:51:23.333507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:23.335350] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:23.335409] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668207, retry interval=1000000, ret=-4179) [2022-08-08 10:51:23.342235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:23.398778] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] clock out of order(cur_ts=1659927083398771, cur_ts_=1659927083360501, delta=38270) [2022-08-08 10:51:23.476199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:23.486567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:23.534507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.535400] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927083535393, cur_ts_=1659927083505553, delta=29840) [2022-08-08 10:51:23.600242] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] [trans sampling] (TRACE=begin_ts=1659927083588061 2022-08-08 02:51:23.588061|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:14417996568521729006, inc:84073741, addr:{ip:"192.168.11.128", port:2882}, t:1659927083587000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=10 ret:0, left_time:10000000, uref:1073741825|[start_task] u=84 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5985-0-0, start_time:1659927083587526, snapshot:1659927083539869|[end_task] u=245 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=12 stc:1659927083588408|[calc_checksum_by_commit] u=25 ret:0, checksum:1404972352, checksum_log_ts:9223372036854775807|[submit_log] u=31 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=11672 trans_version:1659927083588435, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=4 ret:0, log_type:64, log_id:8299235, logging:0, pending:0, uref:1|[end_trans_cb] u=19 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=12123) [2022-08-08 10:51:23.600296] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13134, time_dist: do_cluster_heartbeat=11, generate_cluster_version=13120) [2022-08-08 10:51:23.684320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:23.686102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:23.693251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:23.694186] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:23.694536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:23.695173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:23.719469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:23.761394] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927083761373, cur_ts_=1659927083721442, delta=39931) [2022-08-08 10:51:23.806710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.839466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:23.839590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.855292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.855362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:23.855369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:23.868943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:23.892171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:23.930136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.938568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.961712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:23.961921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:23.962064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.962219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:23.962439] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927083962436, cur_ts_=1659927083938670, delta=23766) [2022-08-08 10:51:23.966222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:23.981254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:23.981384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:23.982545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:23.982874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:23.983971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:23.984068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:23.984012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:24.003288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:24.035343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:24.053662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:24.060060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:24.064160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:24.083683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:24.097846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.100209] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:24.105869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:24.113134] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19122) [2022-08-08 10:51:24.120066] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:24.124177] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31107, time_dist: do_cluster_heartbeat=11, generate_cluster_version=31092) [2022-08-08 10:51:24.135143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.226410] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.231247] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:24.234348] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:24.234373] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:24.240833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:24.247382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:24.270775] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:24.299359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.310860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:24.314035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.328519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5EEE-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.337625] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.337733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4869-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.338407] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4869-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.338493] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4869-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.338538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4869-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.338556] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4869-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.356363] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:24.356400] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668211, retry interval=1000000, ret=-4179) [2022-08-08 10:51:24.364347] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:24.365390] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:24.365439] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668206, retry interval=1000000, ret=-4179) [2022-08-08 10:51:24.396029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.396138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:24.478349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:24.584981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.750483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:24.830252] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659927084830244, cur_ts_=1659927084794041, delta=36203) [2022-08-08 10:51:25.237724] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:25.242819] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:25.242848] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:25.408088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:25.411509] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:25.411578] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668205, retry interval=1000000, ret=-4179) [2022-08-08 10:51:25.413746] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:25.413895] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] task process failed, start retry(max retry time=9223372036854668210, retry interval=1000000, ret=-4179) [2022-08-08 10:51:26.044026] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927086044019, cur_ts_=1659927086022159, delta=21860) [2022-08-08 10:51:26.182676] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] clock out of order(cur_ts=1659927086182668, cur_ts_=1659927086159861, delta=22807) [2022-08-08 10:51:26.245337] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:26.254989] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:26.255046] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:26.265461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B17-0-0] [lt=3521] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.265488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B17-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.266222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B17-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.266401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.266554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.266757] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.266881] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:26.409569] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10428, time_dist: do_cluster_heartbeat=9, generate_cluster_version=10416) [2022-08-08 10:51:26.410750] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927086410743, cur_ts_=1659927086379445, delta=31298) [2022-08-08 10:51:26.446164] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:26.471927] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:26.472068] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668204, retry interval=1000000, ret=-4179) [2022-08-08 10:51:26.476806] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:26.476840] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668209, retry interval=1000000, ret=-4179) [2022-08-08 10:51:26.531394] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1178] [dc=0] clock out of order(cur_ts=1659927086531383, cur_ts_=1659927086482923, delta=48460) [2022-08-08 10:51:26.538904] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=58525) [2022-08-08 10:51:26.595446] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=15] [dc=0] [trans sampling] (TRACE=begin_ts=1659927086593704 2022-08-08 02:51:26.593704|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:16363779126911229838, inc:84073860, addr:{ip:"192.168.11.128", port:2882}, t:1659927086592000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB761-0-0, start_time:1659927086593586, snapshot:1659927086565882|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927086593847|[calc_checksum_by_commit] u=19 ret:0, checksum:3731084562, checksum_log_ts:9223372036854775807|[submit_log] u=36 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1456 trans_version:1659927086592651, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8313842, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=1696) [2022-08-08 10:51:27.257729] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:27.260981] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:27.261016] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:27.479615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C472C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.479705] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C472C-0-0] [lt=73] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.479739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C472C-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.479773] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C472C-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.480030] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C472C-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.480061] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C472C-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.480311] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46EA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:27.539560] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:27.539596] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668203, retry interval=1000000, ret=-4179) [2022-08-08 10:51:27.541953] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:27.541981] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668208, retry interval=1000000, ret=-4179) [2022-08-08 10:51:28.200879] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927088200869, cur_ts_=1659927088171371, delta=29498) [2022-08-08 10:51:28.263287] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:28.267230] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:28.267256] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:28.483044] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927088483031, cur_ts_=1659927088461569, delta=21462) [2022-08-08 10:51:28.584406] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36937) [2022-08-08 10:51:28.584577] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927088584573, cur_ts_=1659927088550114, delta=34459) [2022-08-08 10:51:28.585985] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=39349, time_dist: do_cluster_heartbeat=8, generate_cluster_version=39337) [2022-08-08 10:51:28.596510] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11977) [2022-08-08 10:51:28.599959] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1256] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26995, time_dist: generate_cluster_version=26990) [2022-08-08 10:51:28.605130] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:28.605180] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668207, retry interval=1000000, ret=-4179) [2022-08-08 10:51:28.613358] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:28.615135] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:28.615200] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668202, retry interval=1000000, ret=-4179) [2022-08-08 10:51:29.271761] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:29.275060] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:29.275085] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:29.342813] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927089342806, cur_ts_=1659927089315400, delta=27406) [2022-08-08 10:51:29.657018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:29.662336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4521] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:29.688742] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:51:29.688774] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:51:29.693837] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927089693666, cur_ts_=1659927089672162, delta=21504) [2022-08-08 10:51:29.695435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:29.706806] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:29.706846] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668206, retry interval=1000000, ret=-4179) [2022-08-08 10:51:29.710642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:29.734447] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:29.734503] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668201, retry interval=1000000, ret=-4179) [2022-08-08 10:51:29.734764] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19885) [2022-08-08 10:51:29.815250] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927089815241, cur_ts_=1659927089783883, delta=31358) [2022-08-08 10:51:29.816197] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37969) [2022-08-08 10:51:29.817128] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=2551] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=48220, time_dist: generate_cluster_version=48216) [2022-08-08 10:51:29.821403] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26783, time_dist: generate_cluster_version=26777) [2022-08-08 10:51:29.850956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:29.930746] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927089930728, cur_ts_=1659927089883646, delta=47082) [2022-08-08 10:51:29.994735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:30.023533] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26563) [2022-08-08 10:51:30.025852] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29829, time_dist: do_cluster_heartbeat=10, generate_cluster_version=29814) [2022-08-08 10:51:30.030121] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32865, time_dist: do_cluster_heartbeat=6, generate_cluster_version=32850, cluster_service_self_check=5) [2022-08-08 10:51:30.036141] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:30.064112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.087622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.087845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:30.108252] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:30.118751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=33570] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:30.124806] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14628384}) [2022-08-08 10:51:30.158049] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927090158040, cur_ts_=1659927090131176, delta=26864) [2022-08-08 10:51:30.168271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.174404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=153] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:30.174839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.228401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.228682] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:30.272697] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42905) [2022-08-08 10:51:30.274077] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927090274069, cur_ts_=1659927090241007, delta=33062) [2022-08-08 10:51:30.274096] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45562, time_dist: do_cluster_heartbeat=10, generate_cluster_version=45550) [2022-08-08 10:51:30.286722] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:30.344309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:30.370842] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=98068) [2022-08-08 10:51:30.376961] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927090376952, cur_ts_=1659927090342289, delta=34663) [2022-08-08 10:51:30.377110] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=13] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:16287744280153340658, inc:84073990, addr:"192.168.11.128:2882", t:1659927090206252}, log_id=8313861, used_time=141316) [2022-08-08 10:51:30.396814] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=161731, time_dist: do_cluster_heartbeat=10, generate_cluster_version=161716) [2022-08-08 10:51:30.397388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.432893] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:30.432919] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:30.477481] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=1218] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22028) [2022-08-08 10:51:30.479897] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27795, time_dist: do_cluster_heartbeat=11, generate_cluster_version=27779) [2022-08-08 10:51:30.525972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:30.569892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:30.599618] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927090599610, cur_ts_=1659927090576745, delta=22865) [2022-08-08 10:51:30.611673] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11708) [2022-08-08 10:51:30.622619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:30.622798] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24221, time_dist: generate_cluster_version=24218) [2022-08-08 10:51:30.641883] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927088236345, end_time=1659927090641858, elapsed_time=2405513, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:51:30.642544] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12184) [2022-08-08 10:51:30.642971] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13709, time_dist: do_cluster_heartbeat=170, generate_cluster_version=13536) [2022-08-08 10:51:30.730437] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:30.733300] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=1241] [dc=0] task process failed, start retry(max retry time=9223372036854668205, retry interval=1000000, ret=-4179) [2022-08-08 10:51:30.762385] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:30.762435] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668200, retry interval=1000000, ret=-4179) [2022-08-08 10:51:31.457243] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:31.471174] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=198] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:31.471212] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:31.655989] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927091655981, cur_ts_=1659927091631303, delta=24678) [2022-08-08 10:51:31.759253] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] clock out of order(cur_ts=1659927091759247, cur_ts_=1659927091734595, delta=24652) [2022-08-08 10:51:31.798094] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:31.799385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] task process failed, start retry(max retry time=9223372036854668199, retry interval=1000000, ret=-4179) [2022-08-08 10:51:31.805455] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:31.805496] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668204, retry interval=1000000, ret=-4179) [2022-08-08 10:51:31.838288] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11907, time_dist: do_cluster_heartbeat=12, generate_cluster_version=11891) [2022-08-08 10:51:31.838828] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11629, time_dist: do_cluster_heartbeat=6, generate_cluster_version=11620) [2022-08-08 10:51:32.450789] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927092450781, cur_ts_=1659927092426100, delta=24681) [2022-08-08 10:51:32.475972] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:32.479174] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:32.479249] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:32.624401] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:32.640329] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] [trans sampling] (TRACE=begin_ts=1659927092639352 2022-08-08 02:51:32.639352|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:8632721467256677487, inc:84074086, addr:{ip:"192.168.11.128", port:2882}, t:1659927092639000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A59B3-0-0, start_time:1659927092639233, snapshot:1659927092545952|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927092639494|[calc_checksum_by_commit] u=22 ret:0, checksum:4248411880, checksum_log_ts:9223372036854775807|[submit_log] u=28 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=693 trans_version:1659927092639517, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8299281, logging:0, pending:0, uref:1|[end_trans_cb] u=22 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=938) [2022-08-08 10:51:32.830469] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:32.830509] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668198, retry interval=1000000, ret=-4179) [2022-08-08 10:51:32.840171] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:32.840204] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668203, retry interval=1000000, ret=-4179) [2022-08-08 10:51:32.927220] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927092927214, cur_ts_=1659927092905625, delta=21589) [2022-08-08 10:51:33.366958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:33.481831] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:33.484730] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:33.484780] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:33.498306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:33.507937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:33.552137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.664130] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927093664124, cur_ts_=1659927093638301, delta=25823) [2022-08-08 10:51:33.666866] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26135, time_dist: do_cluster_heartbeat=9, generate_cluster_version=26122) [2022-08-08 10:51:33.705089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:33.705152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:33.712127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:33.712210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:33.715491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:33.717437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:33.736432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:33.828634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.855960] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10654) [2022-08-08 10:51:33.856585] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12133, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12121) [2022-08-08 10:51:33.860396] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:33.860447] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668197, retry interval=1000000, ret=-4179) [2022-08-08 10:51:33.861260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.861434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:33.863647] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:33.863675] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668202, retry interval=1000000, ret=-4179) [2022-08-08 10:51:33.871256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:33.871606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:33.880718] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.890532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:33.907877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:33.945643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.956289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.981811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:33.981933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:33.986763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:33.987327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=187] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:33.987367] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:33.996972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:33.997907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:33.998680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:33.999896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:33.999935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:34.006107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:34.007184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.018900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:34.057753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:34.071140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:34.081583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:34.082345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:34.106858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:34.116990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.117137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:34.128894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:34.137818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:34.151176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.245163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.262704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:34.262875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:34.315087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.328135] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:34.335522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:34.338648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.356166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.415301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.417522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:34.487247] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:34.490488] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:34.490525] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:34.500126] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:34.610279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.771920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:34.891831] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:34.892367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:34.894382] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:34.896221] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=345] [dc=0] task process failed, start retry(max retry time=9223372036854668201, retry interval=1000000, ret=-4179) [2022-08-08 10:51:34.921534] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1830] [dc=0] clock out of order(cur_ts=1659927094921528, cur_ts_=1659927094894139, delta=27389) [2022-08-08 10:51:34.921588] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:34.921772] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854668196, retry interval=1000000, ret=-4179) [2022-08-08 10:51:34.923542] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27011, time_dist: generate_cluster_version=27008) [2022-08-08 10:51:35.185794] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:51:35.428424] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11136) [2022-08-08 10:51:35.492635] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:35.496645] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:35.496670] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:35.653399] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927095652511 2022-08-08 02:51:35.652511|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:4726174593926851275, inc:84074206, addr:{ip:"192.168.11.128", port:2882}, t:1659927095651000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A59C2-0-0, start_time:1659927095652392, snapshot:1659927095554146|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927095652653|[calc_checksum_by_commit] u=26 ret:0, checksum:2099238489, checksum_log_ts:9223372036854775807|[submit_log] u=85 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=543 trans_version:1659927095652677, uref:1073741825|[exiting] u=23 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8299296, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=846) [2022-08-08 10:51:35.920094] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:35.920198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:35.921761] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:35.921841] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668200, retry interval=1000000, ret=-4179) [2022-08-08 10:51:35.945160] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:35.945199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:35.946476] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:35.946544] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668195, retry interval=1000000, ret=-4179) [2022-08-08 10:51:36.229204] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927096229199, cur_ts_=1659927096196481, delta=32718) [2022-08-08 10:51:36.254420] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12539, time_dist: generate_cluster_version=12534) [2022-08-08 10:51:36.420829] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B35-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:36.500806] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:36.501766] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927096501761, cur_ts_=1659927096470245, delta=31516) [2022-08-08 10:51:36.508274] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:36.508300] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:36.887029] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15537) [2022-08-08 10:51:36.887363] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16631, time_dist: generate_cluster_version=16599) [2022-08-08 10:51:36.948210] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927096948200, cur_ts_=1659927096916743, delta=31457) [2022-08-08 10:51:37.101459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:37.101590] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:37.125177] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927097125169, cur_ts_=1659927097103150, delta=22019) [2022-08-08 10:51:37.127637] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:37.127675] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668199, retry interval=1000000, ret=-4179) [2022-08-08 10:51:37.140398] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:37.140438] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668194, retry interval=1000000, ret=-4179) [2022-08-08 10:51:37.140587] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13591) [2022-08-08 10:51:37.141704] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18667, time_dist: generate_cluster_version=18657, cluster_service_self_check=5) [2022-08-08 10:51:37.460868] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=280] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24015) [2022-08-08 10:51:37.461102] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25126, time_dist: generate_cluster_version=25124) [2022-08-08 10:51:37.510701] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927097510691, cur_ts_=1659927097486129, delta=24562) [2022-08-08 10:51:37.515346] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:37.524767] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:37.524796] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:37.886456] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927097886447, cur_ts_=1659927097847146, delta=39301) [2022-08-08 10:51:38.134124] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.134147] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.134333] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.136521] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.136561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.137122] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.137196] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.137319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C46F9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.168184] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:38.168220] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668198, retry interval=1000000, ret=-4179) [2022-08-08 10:51:38.190144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.191011] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:38.192295] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:38.192357] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668193, retry interval=1000000, ret=-4179) [2022-08-08 10:51:38.285742] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=49366) [2022-08-08 10:51:38.285969] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=50361, time_dist: do_cluster_heartbeat=11, generate_cluster_version=50348) [2022-08-08 10:51:38.434830] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927098434821, cur_ts_=1659927098410597, delta=24224) [2022-08-08 10:51:38.449505] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13013) [2022-08-08 10:51:38.450357] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16967, time_dist: do_cluster_heartbeat=12, generate_cluster_version=16952) [2022-08-08 10:51:38.468945] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19378) [2022-08-08 10:51:38.469234] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21660, time_dist: do_cluster_heartbeat=32, generate_cluster_version=21626) [2022-08-08 10:51:38.538452] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:38.548925] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:38.549019] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:38.677729] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] [trans sampling] (TRACE=begin_ts=1659927098675230 2022-08-08 02:51:38.675230|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17838331908467125581, inc:84074317, addr:{ip:"192.168.11.128", port:2882}, t:1659927098674000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A59D1-0-0, start_time:1659927098674854, snapshot:1659927098586692|[end_task] u=253 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927098675503|[calc_checksum_by_commit] u=21 ret:0, checksum:1441823745, checksum_log_ts:9223372036854775807|[submit_log] u=47 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2043 trans_version:1659927098675526, uref:1073741825|[exiting] u=26 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299311, logging:0, pending:0, uref:1|[end_trans_cb] u=33 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=2454) [2022-08-08 10:51:39.189911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:39.190037] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:39.190537] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:39.190587] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668197, retry interval=1000000, ret=-4179) [2022-08-08 10:51:39.208657] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17408) [2022-08-08 10:51:39.216199] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:39.217697] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=150] [dc=0] task process failed, start retry(max retry time=9223372036854668192, retry interval=1000000, ret=-4179) [2022-08-08 10:51:39.555020] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:39.557983] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:39.558018] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:39.650401] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=400] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10305) [2022-08-08 10:51:39.654016] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14739, time_dist: do_cluster_heartbeat=13, generate_cluster_version=14721) [2022-08-08 10:51:39.660534] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10022) [2022-08-08 10:51:39.660907] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11379, time_dist: generate_cluster_version=11376) [2022-08-08 10:51:39.662025] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:51:39.662156] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:51:39.713384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:39.735567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:39.835620] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] clock out of order(cur_ts=1659927099835614, cur_ts_=1659927099810466, delta=25148) [2022-08-08 10:51:39.868019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:39.967059] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927099967031, cur_ts_=1659927099932604, delta=34427) [2022-08-08 10:51:40.040734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:40.084291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:40.095358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.096987] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927100096982, cur_ts_=1659927100043226, delta=53756) [2022-08-08 10:51:40.123125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.123187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:40.126123] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31318) [2022-08-08 10:51:40.126711] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33342, time_dist: do_cluster_heartbeat=9, generate_cluster_version=33329) [2022-08-08 10:51:40.127340] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27854, time_dist: do_cluster_heartbeat=11, generate_cluster_version=27841) [2022-08-08 10:51:40.133484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:40.154179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=754] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:40.181673] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14632768}) [2022-08-08 10:51:40.183429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.195385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.197874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:40.249527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.251587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:40.259385] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:40.259420] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668196, retry interval=1000000, ret=-4179) [2022-08-08 10:51:40.275241] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14071) [2022-08-08 10:51:40.276930] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:40.276980] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668191, retry interval=1000000, ret=-4179) [2022-08-08 10:51:40.364642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:40.405828] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927098236513, end_time=1659927100405818, elapsed_time=2169305, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:51:40.417434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.469136] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927100469128, cur_ts_=1659927100441799, delta=27329) [2022-08-08 10:51:40.546976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:40.559888] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:40.563356] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:40.563392] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:40.595127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:40.640074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:40.770436] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=147] [dc=0] clock out of order(cur_ts=1659927100769881, cur_ts_=1659927100740234, delta=29647) [2022-08-08 10:51:41.262183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4881-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:41.262357] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4881-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:41.270486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:41.284902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:41.284932] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:41.288939] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:41.288995] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668195, retry interval=1000000, ret=-4179) [2022-08-08 10:51:41.297527] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:41.299087] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:41.299147] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668190, retry interval=1000000, ret=-4179) [2022-08-08 10:51:41.494813] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:41.495195] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=356] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:41.565640] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:41.570087] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:41.570124] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:41.700422] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] [trans sampling] (TRACE=begin_ts=1659927101699564 2022-08-08 02:51:41.699564|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:6606345764772373229, inc:84074429, addr:{ip:"192.168.11.128", port:2882}, t:1659927101698000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=8 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB7AE-0-0, start_time:1659927101699378, snapshot:1659927101695592|[end_task] u=135 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927101699718|[calc_checksum_by_commit] u=18 ret:0, checksum:453652548, checksum_log_ts:9223372036854775807|[submit_log] u=92 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=516 trans_version:1659927101699526, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8313919, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=818) [2022-08-08 10:51:41.797855] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1158] [dc=0] clock out of order(cur_ts=1659927101797846, cur_ts_=1659927101763991, delta=33855) [2022-08-08 10:51:42.049480] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927102049469, cur_ts_=1659927102016491, delta=32978) [2022-08-08 10:51:42.313647] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:42.313670] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:42.315646] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:42.315674] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:42.357256] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927102357171, cur_ts_=1659927102324799, delta=32372) [2022-08-08 10:51:42.372150] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=220] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13247) [2022-08-08 10:51:42.375283] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17876, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17862) [2022-08-08 10:51:42.381316] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24810, time_dist: print_stat=9, do_cluster_heartbeat=3, generate_cluster_version=24793) [2022-08-08 10:51:42.401747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:42.406224] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:42.406472] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] task process failed, start retry(max retry time=9223372036854668189, retry interval=1000000, ret=-4179) [2022-08-08 10:51:42.422876] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15788) [2022-08-08 10:51:42.423741] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:42.434150] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:42.434207] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668194, retry interval=1000000, ret=-4179) [2022-08-08 10:51:42.557497] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927102557489, cur_ts_=1659927102514772, delta=42717) [2022-08-08 10:51:42.578036] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:42.578362] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19126) [2022-08-08 10:51:42.578734] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21220, time_dist: do_cluster_heartbeat=10, generate_cluster_version=21208) [2022-08-08 10:51:42.587601] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:42.587664] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:43.245523] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927103245516, cur_ts_=1659927103215234, delta=30282) [2022-08-08 10:51:43.390524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:43.427144] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927103427140, cur_ts_=1659927103405565, delta=21575) [2022-08-08 10:51:43.430541] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4884-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:43.430573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4884-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:43.487702] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:43.487782] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668188, retry interval=1000000, ret=-4179) [2022-08-08 10:51:43.493269] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:43.493356] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668193, retry interval=1000000, ret=-4179) [2022-08-08 10:51:43.515076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:43.527273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:43.564474] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:43.564619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:43.564795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:43.564870] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:43.570347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:43.595553] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:43.600159] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:43.600199] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:43.727731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:43.738601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:43.738679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:43.738766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:43.739064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:43.740881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:43.740920] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] clock out of order(cur_ts=1659927103740917, cur_ts_=1659927103705359, delta=35558) [2022-08-08 10:51:43.783233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:43.836246] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10632][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511628002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8313929, proposal_id:{time_to_usec:1659815620290075, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:47373951, size:568}, after_consume_timestamp:1659927103789589}, end_time=1659927103825225, begin_time=1659927103789589, cost_time=35636) [2022-08-08 10:51:43.837729] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24230, time_dist: do_cluster_heartbeat=9, generate_cluster_version=24217) [2022-08-08 10:51:43.847249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:43.855710] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30249) [2022-08-08 10:51:43.857090] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=66645, time_dist: generate_cluster_version=66641) [2022-08-08 10:51:43.885917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:43.886021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:43.887827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:43.896184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:43.897979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=405] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:43.909086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:43.930559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:43.964204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:43.972376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.003364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:44.003392] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.004112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.004322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.005742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:44.013786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:44.014175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:44.016150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:44.016208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:44.016230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:44.025408] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.026656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.035533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:44.078237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:44.094552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.100383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.110536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.126248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:44.137246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.143602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:44.145683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:44.161201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:44.174571] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.216324] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] clock out of order(cur_ts=1659927104216317, cur_ts_=1659927104185804, delta=30513) [2022-08-08 10:51:44.267175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.281595] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:44.285876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:44.333218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.354454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:44.354505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.377001] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.408140] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:44.434400] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2190] [dc=0] clock out of order(cur_ts=1659927104434392, cur_ts_=1659927104413908, delta=20484) [2022-08-08 10:51:44.434622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:44.434767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.520332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:44.522957] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:44.523007] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668187, retry interval=1000000, ret=-4179) [2022-08-08 10:51:44.527539] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:44.527572] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668192, retry interval=1000000, ret=-4179) [2022-08-08 10:51:44.605389] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:44.608675] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:44.608721] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:44.626217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.659629] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927104659620, cur_ts_=1659927104636916, delta=22704) [2022-08-08 10:51:44.721772] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927104718505 2022-08-08 02:51:44.718505|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:9233877331187230528, inc:84074547, addr:{ip:"192.168.11.128", port:2882}, t:1659927104716000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A59EF-0-0, start_time:1659927104718337, snapshot:1659927104663522|[end_task] u=128 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927104718650|[calc_checksum_by_commit] u=21 ret:0, checksum:3957422025, checksum_log_ts:9223372036854775807|[submit_log] u=156 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2801 trans_version:1659927104718671, uref:1073741825|[exiting] u=27 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299341, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=3178) [2022-08-08 10:51:44.790867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=431] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:44.846127] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927104846122, cur_ts_=1659927104820748, delta=25374) [2022-08-08 10:51:45.550608] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:45.550647] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668186, retry interval=1000000, ret=-4179) [2022-08-08 10:51:45.560827] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:45.560861] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668191, retry interval=1000000, ret=-4179) [2022-08-08 10:51:45.562535] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13610, time_dist: do_cluster_heartbeat=9, generate_cluster_version=13598) [2022-08-08 10:51:45.612968] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:45.654523] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927105654514, cur_ts_=1659927105619383, delta=35131) [2022-08-08 10:51:45.720333] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:45.720397] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:45.760985] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2588] [dc=0] clock out of order(cur_ts=1659927105760976, cur_ts_=1659927105731491, delta=29485) [2022-08-08 10:51:45.905069] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19916) [2022-08-08 10:51:45.905711] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33849, time_dist: do_cluster_heartbeat=19, generate_cluster_version=33827) [2022-08-08 10:51:45.906473] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27495, time_dist: do_cluster_heartbeat=10, generate_cluster_version=27482) [2022-08-08 10:51:46.190839] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12610, time_dist: generate_cluster_version=12601, cluster_service_self_check=5) [2022-08-08 10:51:46.585792] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:46.585829] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668185, retry interval=1000000, ret=-4179) [2022-08-08 10:51:46.586628] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:46.586654] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668190, retry interval=1000000, ret=-4179) [2022-08-08 10:51:46.860745] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:46.863978] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:46.864014] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:46.992331] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927106991487, cur_ts_=1659927106964069, delta=27418) [2022-08-08 10:51:47.632626] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927107632617, cur_ts_=1659927107608307, delta=24310) [2022-08-08 10:51:47.647820] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=221] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:47.647888] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668189, retry interval=1000000, ret=-4179) [2022-08-08 10:51:47.649456] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:47.649487] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668184, retry interval=1000000, ret=-4179) [2022-08-08 10:51:47.866630] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:47.871137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:47.871176] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:48.602539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B59-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:48.602707] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B59-0-0] [lt=146] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:48.602841] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:48.602861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:48.602874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:51:48.672836] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:48.672887] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668188, retry interval=1000000, ret=-4179) [2022-08-08 10:51:48.685725] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:48.687225] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854668183, retry interval=1000000, ret=-4179) [2022-08-08 10:51:48.873981] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:48.877128] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:48.877166] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:49.145938] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:51:49.647903] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:51:49.647924] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:51:49.678576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F12-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.678899] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F12-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.679118] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F12-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.679248] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F12-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.691497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C488D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.691577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C488D-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.704134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.704522] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.705937] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:49.705994] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668187, retry interval=1000000, ret=-4179) [2022-08-08 10:51:49.710245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:49.712091] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:49.712125] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668182, retry interval=1000000, ret=-4179) [2022-08-08 10:51:49.733586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:49.755869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:49.794264] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14637088}) [2022-08-08 10:51:49.879098] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:49.883909] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:49.883936] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:49.891426] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:49.937897] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11113) [2022-08-08 10:51:49.938417] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12504, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12492) [2022-08-08 10:51:49.976594] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927108237649, end_time=1659927109976522, elapsed_time=1738873, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:51:50.060979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:50.109037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=8433] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:50.112822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.143921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:50.144040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.154234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:50.176377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:50.209622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.219912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.220313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=9739] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:50.270962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:50.273288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.386795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:50.438474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.524002] WARN [STORAGE.TRANS] end_trans (ob_trans_service.cpp:1063) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=27] [dc=0] transaction timeout and already terminated(trans_desc={tenant_id:1001, trans_id:{hash:14122346763008492130, inc:84066165, addr:"192.168.11.128:2882", t:1659926903641018}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:0, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}) [2022-08-08 10:51:50.524086] WARN [STORAGE.TRANS] end_trans (ob_trans_service.cpp:1148) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=36] [dc=0] end transaction error(ret=-6210, ret="OB_TRANS_TIMEOUT", trans_desc={tenant_id:1001, trans_id:{hash:0, inc:0, addr:"192.168.11.128", t:0}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:1, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}, is_rollback=false) [2022-08-08 10:51:50.524173] WARN [STORAGE.TRANS] end_trans (ob_trans_service.cpp:1155) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=21] [dc=0] end transaction failed(ret=-6210, ret="OB_TRANS_TIMEOUT", trans_desc={tenant_id:1001, trans_id:{hash:0, inc:0, addr:"192.168.11.128", t:0}, snapshot_version:1659926987012420, trans_snapshot_version:-1, trans_param:[access_mode=1, type=3, isolation=1, magic=17361641481138401520, autocommit=0, consistency_type=0(CURRENT_READ), read_snapshot_type=1(STATEMENT_SNAPSHOT), cluster_version=12885032962, is_inner_trans=0], participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], stmt_participants:[{tid:1100611139453778, partition_id:0, part_cnt:0}], sql_no:25769803782, max_sql_no:25769803782, cur_stmt_desc:[stmt_tenant_id=1001, phy_plan_type=1, stmt_type=5, consistency_level=0, execution_id=86691235, sql_id=, trace_id=YB42C0A80B80-0005E3CF9BC835E6-0-0, is_inner_sql=0, app_trace_id_str=, cur_stmt_specified_snapshot_version=-1, cur_query_start_time=1659927028571834, is_sfu=0, is_contain_inner_table=0, trx_lock_timeout=-1, snapshot_use_publish_version=1, skip_trx=0], need_rollback:false, trans_expired_time:1659927003596428, cur_stmt_expired_time:1659927038501834, trans_type:2, is_all_select_stmt:false, stmt_participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, participants_pla:{partitions:[{tid:1100611139453778, partition_id:0, part_cnt:0}], leaders:["192.168.11.128:2882"], type_array:[0]}, is_sp_trans_exiting:false, sql_trans_action:1, snapshot_gene_type:1, local_consistency_type:-1, snapshot_gene_type_str:"APPOINT", local_consistency_type_str:"UNKNOWN", session_id:3221573953, proxy_session_id:0, app_trace_id_confirmed:true, can_elr:false, is_dup_table_trans:false, is_local_trans:true, trans_need_wait_wrap:{receive_gts_ts:{mts:0}, need_wait_interval_us:0}, is_fast_select:false, trace_info:{app_trace_info:"", app_trace_id:""}, standalone_stmt_desc:{this:0x7f69961f3aa8, trans_id:{hash:14431837840091723769, inc:84063949, addr:"192.168.11.128:2882", t:1659926862922908}, tenant_id:0, stmt_expired_time:-1, trx_lock_timeout:-1, consistency_type:-1, read_snapshot_type:-1, snapshot_version:-1, is_local_single_partition:false, is_standalone_stmt_end:true, first_pkey:{tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}}, is_nested_stmt:false, stmt_min_sql_no:25769803782, stmt_min_seq_no:1659926986970265, curr_msg_seq:0, xid:{gtrid_str:"", bqual_str:"", format_id:1, gtrid_str_.ptr():"data_size:0, data:", bqual_str_.ptr():"data_size:0, data:"}, gc_participants:[], trans_status_registered:true}) [2022-08-08 10:51:50.524201] WARN [SQL.EXE] end_trans (ob_sql_trans_control.cpp:630) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=25] [dc=0] fail end trans(ret=-6210, callback_type=1) [2022-08-08 10:51:50.524207] WARN [SQL.EXE] explicit_end_trans (ob_sql_trans_control.cpp:445) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=5] [dc=0] fail end explicit trans(is_rollback=false, ret=-6210) [2022-08-08 10:51:50.524211] WARN [SQL.ENG] end_trans (ob_tcl_executor.cpp:63) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=3] [dc=0] fail end trans(ret=-6210) [2022-08-08 10:51:50.524215] WARN [SQL] open_cmd (ob_result_set.cpp:78) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=3] [dc=0] execute cmd failed(ret=-6210) [2022-08-08 10:51:50.524219] WARN [SQL.SESSION] sync_open (ob_result_set.cpp:140) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=3] [dc=0] fail to exec execute()(ret=-6210) [2022-08-08 10:51:50.524225] WARN [SERVER] response_result (ob_async_cmd_driver.cpp:60) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=4] [dc=0] close result set fail(close_ret=-6210) [2022-08-08 10:51:50.524230] WARN [SERVER] response_result (ob_async_cmd_driver.cpp:68) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=5] [dc=0] result set open failed, async end trans submmited, don't retry(ret=-6210) [2022-08-08 10:51:50.524233] WARN [SERVER] response_result (obmp_query.cpp:1290) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=3] [dc=0] fail response async result(ret=-6210) [2022-08-08 10:51:50.524238] WARN [SERVER] do_process (obmp_query.cpp:775) [11024][0][YB42C0A80B80-0005E3CF9BC835E7-0-0] [lt=4] [dc=0] execute query fail(ret=-6210, timeout_timestamp=1659927120523839) [2022-08-08 10:51:50.579115] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927110579108, cur_ts_=1659927110545345, delta=33763) [2022-08-08 10:51:50.583471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:50.624146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:50.658325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:50.732637] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:50.732691] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668186, retry interval=1000000, ret=-4179) [2022-08-08 10:51:50.738546] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:50.740000] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=137] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:50.740066] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668181, retry interval=1000000, ret=-4179) [2022-08-08 10:51:50.744035] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] [trans sampling] (TRACE=begin_ts=1659927110740500 2022-08-08 02:51:50.740500|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:8523199064750397454, inc:84074768, addr:{ip:"192.168.11.128", port:2882}, t:1659927110739000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7A4B-0-0, start_time:1659927110740369, snapshot:1659927110732882|[end_task] u=101 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927110740618|[calc_checksum_by_commit] u=18 ret:0, checksum:3003390620, checksum_log_ts:9223372036854775807|[submit_log] u=59 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3257 trans_version:1659927110739917, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987156, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=3493) [2022-08-08 10:51:50.890758] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:50.894343] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:50.894379] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:51.280501] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927111280497, cur_ts_=1659927111248506, delta=31991) [2022-08-08 10:51:51.413632] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] clock out of order(cur_ts=1659927111413624, cur_ts_=1659927111366758, delta=46866) [2022-08-08 10:51:51.416411] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19209, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19196) [2022-08-08 10:51:51.417021] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10525, time_dist: do_cluster_heartbeat=7, generate_cluster_version=10515) [2022-08-08 10:51:51.735337] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F15-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:51.735517] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F15-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:51.764467] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:51.765891] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=171] [dc=0] task process failed, start retry(max retry time=9223372036854668185, retry interval=1000000, ret=-4179) [2022-08-08 10:51:51.766543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:51.766574] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:51.767863] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:51.767909] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668180, retry interval=1000000, ret=-4179) [2022-08-08 10:51:51.897251] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:51.900452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:51.900505] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:52.508032] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=252] [dc=0] clock out of order(cur_ts=1659927112508027, cur_ts_=1659927112477946, delta=30081) [2022-08-08 10:51:52.625844] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=1136] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:52.768254] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4710-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:52.790798] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:52.791009] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] task process failed, start retry(max retry time=9223372036854668179, retry interval=1000000, ret=-4179) [2022-08-08 10:51:52.794077] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=147] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:52.795686] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:52.795749] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668184, retry interval=1000000, ret=-4179) [2022-08-08 10:51:52.902807] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:52.909191] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:52.909248] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:53.127820] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927113127810, cur_ts_=1659927113101692, delta=26118) [2022-08-08 10:51:53.405681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=625] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:53.535134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=186] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:53.552665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:53.593699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:53.749918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=10832] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:53.759839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:53.759900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:53.760043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:53.760107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:53.761516] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:53.762496] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927113761477 2022-08-08 02:51:53.761477|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:14511576098245667770, inc:84074878, addr:{ip:"192.168.11.128", port:2882}, t:1659927113760000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=11 ret:0, left_time:10000000, uref:1073741825|[start_task] u=10 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5A1D-0-0, start_time:1659927113760771, snapshot:1659927113730003|[end_task] u=164 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927113761669|[calc_checksum_by_commit] u=69 ret:0, checksum:1623681336, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=653 trans_version:1659927113761739, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299387, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=981) [2022-08-08 10:51:53.797706] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4893-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:53.811704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:53.814678] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:53.815991] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:53.816054] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668178, retry interval=1000000, ret=-4179) [2022-08-08 10:51:53.824480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:53.826447] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:53.826497] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668183, retry interval=1000000, ret=-4179) [2022-08-08 10:51:53.872534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:53.905549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:53.911160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:53.911562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=4636] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:53.914929] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:53.916180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:53.928973] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:53.930023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:53.932736] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:53.932879] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:53.947857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:53.989155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:53.995161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.021710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:54.022697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.022796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:51:54.022889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.025753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.036525] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:51:54.036623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:54.037260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:54.038765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:54.041070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:54.043678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.046506] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.056901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:54.099975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:54.109952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.116776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.126847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.146157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:54.163543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:54.163435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.174504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:54.177446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:51:54.199241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=125] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.283055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.303470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:51:54.304515] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:54.356485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.378915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.379187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:54.392166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.454716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:51:54.455146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.505464] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:54.537779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:54.649505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.816013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:51:54.842348] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:54.843767] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:54.843895] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668177, retry interval=1000000, ret=-4179) [2022-08-08 10:51:54.850074] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:54.850549] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] task process failed, start retry(max retry time=9223372036854668182, retry interval=1000000, ret=-4179) [2022-08-08 10:51:54.941354] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:54.942017] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927114942014, cur_ts_=1659927114921857, delta=20157) [2022-08-08 10:51:54.947637] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:54.947666] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:55.135969] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927115135964, cur_ts_=1659927115110845, delta=25119) [2022-08-08 10:51:55.215852] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:51:55.882920] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:55.882973] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668181, retry interval=1000000, ret=-4179) [2022-08-08 10:51:55.889126] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:55.889203] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668176, retry interval=1000000, ret=-4179) [2022-08-08 10:51:55.950786] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:55.954321] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:55.954344] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:56.256295] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927116256289, cur_ts_=1659927116228947, delta=27342) [2022-08-08 10:51:56.821530] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=196] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10510) [2022-08-08 10:51:56.821927] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927116807889 2022-08-08 02:51:56.807889|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:9827762753351039093, inc:84074994, addr:{ip:"192.168.11.128", port:2882}, t:1659927116802000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5A2C-0-0, start_time:1659927116807702, snapshot:1659927116765029|[end_task] u=141 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927116808049|[calc_checksum_by_commit] u=23 ret:0, checksum:2869174825, checksum_log_ts:9223372036854775807|[submit_log] u=59 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=13633 trans_version:1659927116808074, uref:1073741825|[exiting] u=24 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=48 ret:0, log_type:64, log_id:8299402, logging:0, pending:0, uref:1|[end_trans_cb] u=20 ret:0, param:0, param:true|[destroy] u=7 arg1:140085232375600|total_timeu=13978) [2022-08-08 10:51:56.822096] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14786, time_dist: generate_cluster_version=14784) [2022-08-08 10:51:56.919212] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927116919206, cur_ts_=1659927116883355, delta=35851) [2022-08-08 10:51:56.938732] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:56.938766] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668175, retry interval=1000000, ret=-4179) [2022-08-08 10:51:56.939539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:56.939655] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:56.942104] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:56.942775] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668180, retry interval=1000000, ret=-4179) [2022-08-08 10:51:56.959655] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:56.967778] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:56.967816] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:57.940886] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4899-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:57.941046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4899-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:57.963545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:57.963662] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:51:57.965109] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:57.965146] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668179, retry interval=1000000, ret=-4179) [2022-08-08 10:51:57.967001] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:57.967087] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] task process failed, start retry(max retry time=9223372036854668174, retry interval=1000000, ret=-4179) [2022-08-08 10:51:57.980866] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:57.984578] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:57.984629] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:58.182085] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927118182079, cur_ts_=1659927118161451, delta=20628) [2022-08-08 10:51:58.276642] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24073) [2022-08-08 10:51:58.307012] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=60794, time_dist: do_cluster_heartbeat=12, generate_cluster_version=60778) [2022-08-08 10:51:58.315330] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=38624) [2022-08-08 10:51:58.325756] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=78471, time_dist: generate_cluster_version=78467) [2022-08-08 10:51:58.432101] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16524) [2022-08-08 10:51:58.432815] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18182, time_dist: do_cluster_heartbeat=10, generate_cluster_version=18169) [2022-08-08 10:51:58.455734] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18721) [2022-08-08 10:51:58.456024] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20274, time_dist: do_cluster_heartbeat=9, generate_cluster_version=20262) [2022-08-08 10:51:58.514838] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927118514830, cur_ts_=1659927118484518, delta=30312) [2022-08-08 10:51:58.874557] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] clock out of order(cur_ts=1659927118874552, cur_ts_=1659927118843395, delta=31157) [2022-08-08 10:51:58.884470] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10680, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10665) [2022-08-08 10:51:58.989097] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:51:59.034778] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:51:59.035016] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:51:59.041872] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33608) [2022-08-08 10:51:59.060047] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=54545, time_dist: generate_cluster_version=54542) [2022-08-08 10:51:59.064350] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:59.064391] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668173, retry interval=1000000, ret=-4179) [2022-08-08 10:51:59.065836] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:51:59.065886] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668178, retry interval=1000000, ret=-4179) [2022-08-08 10:51:59.404072] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927119404065, cur_ts_=1659927119383541, delta=20524) [2022-08-08 10:51:59.644165] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:51:59.644197] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:51:59.756083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:59.773560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:59.834855] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927119833917 2022-08-08 02:51:59.833917|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:8833877474555461745, inc:84075107, addr:{ip:"192.168.11.128", port:2882}, t:1659927119832000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5A3B-0-0, start_time:1659927119833730, snapshot:1659927119805680|[end_task] u=138 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927119834073|[calc_checksum_by_commit] u=22 ret:0, checksum:807229677, checksum_log_ts:9223372036854775807|[submit_log] u=29 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=639 trans_version:1659927119834097, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8299417, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=886) [2022-08-08 10:51:59.908587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:51:59.923504] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927119923496, cur_ts_=1659927119902397, delta=21099) [2022-08-08 10:52:00.047700] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17716) [2022-08-08 10:52:00.047966] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:00.048001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x6d3369c 0x3cd322f 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:00.048011] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14641408}) [2022-08-08 10:52:00.057649] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:00.059746] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=45653, time_dist: generate_cluster_version=45650) [2022-08-08 10:52:00.066352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C489C-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:00.066394] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C489C-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:00.086548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:00.122757] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:00.123186] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668172, retry interval=1000000, ret=-4179) [2022-08-08 10:52:00.131276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:00.155510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=212] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.174556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:00.177973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:00.175420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.188518] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26671) [2022-08-08 10:52:00.198796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:00.222321] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927120222311, cur_ts_=1659927120199001, delta=23310) [2022-08-08 10:52:00.226326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.228251] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11103) [2022-08-08 10:52:00.228892] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12637, time_dist: do_cluster_heartbeat=8, generate_cluster_version=12625) [2022-08-08 10:52:00.240004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:00.260466] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:00.260488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:00.289158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:00.289275] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:00.289326] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668177, retry interval=1000000, ret=-4179) [2022-08-08 10:52:00.241488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.293893] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=64101) [2022-08-08 10:52:00.294324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.329514] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=104611, time_dist: do_cluster_heartbeat=11, generate_cluster_version=104594) [2022-08-08 10:52:00.340363] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46410) [2022-08-08 10:52:00.404956] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927120404951, cur_ts_=1659927120376041, delta=28910) [2022-08-08 10:52:00.421885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:00.457210] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=40407) [2022-08-08 10:52:00.463993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.464637] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=60210, time_dist: do_cluster_heartbeat=10, generate_cluster_version=60197) [2022-08-08 10:52:00.472114] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14843) [2022-08-08 10:52:00.474885] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=57318, time_dist: do_cluster_heartbeat=10, generate_cluster_version=57299, cluster_service_self_check=6) [2022-08-08 10:52:00.544977] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927118249460, end_time=1659927120544963, elapsed_time=2295503, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:00.602999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:00.645650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:00.675983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:01.063045] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:01.067337] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:01.067497] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:01.149562] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=360] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:01.149619] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668171, retry interval=1000000, ret=-4179) [2022-08-08 10:52:01.221109] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927121221100, cur_ts_=1659927121194463, delta=26637) [2022-08-08 10:52:01.321685] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:01.322053] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=242] [dc=0] task process failed, start retry(max retry time=9223372036854668176, retry interval=1000000, ret=-4179) [2022-08-08 10:52:01.676151] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927121676146, cur_ts_=1659927121646813, delta=29333) [2022-08-08 10:52:01.955610] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927121955601, cur_ts_=1659927121918878, delta=36723) [2022-08-08 10:52:02.071812] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:02.073581] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14321) [2022-08-08 10:52:02.076427] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28620, time_dist: do_cluster_heartbeat=5, generate_cluster_version=28611) [2022-08-08 10:52:02.081324] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11784, time_dist: generate_cluster_version=11779) [2022-08-08 10:52:02.083716] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:02.083787] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:02.201037] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:02.201080] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:02.202748] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:02.202822] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668170, retry interval=1000000, ret=-4179) [2022-08-08 10:52:02.324490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C489F-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:02.324540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C489F-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:02.343119] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:02.343201] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668175, retry interval=1000000, ret=-4179) [2022-08-08 10:52:02.479131] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13345, time_dist: generate_cluster_version=13341) [2022-08-08 10:52:02.479616] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10463, time_dist: generate_cluster_version=10461) [2022-08-08 10:52:02.712991] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:02.713045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:02.775850] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2073] [dc=0] clock out of order(cur_ts=1659927122775841, cur_ts_=1659927122753456, delta=22385) [2022-08-08 10:52:02.851581] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927122849850 2022-08-08 02:52:02.849850|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:8051562028789593528, inc:84075228, addr:{ip:"192.168.11.128", port:2882}, t:1659927122849000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB818-0-0, start_time:1659927122849742, snapshot:1659927122780780|[end_task] u=123 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927122849990|[calc_checksum_by_commit] u=18 ret:0, checksum:242456760, checksum_log_ts:9223372036854775807|[submit_log] u=333 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1159 trans_version:1659927122849283, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8314025, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=1676) [2022-08-08 10:52:02.865167] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13682) [2022-08-08 10:52:02.865380] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14956, time_dist: do_cluster_heartbeat=6, generate_cluster_version=14948) [2022-08-08 10:52:03.093647] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:03.098002] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:03.098045] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:03.151106] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=371] [dc=0] clock out of order(cur_ts=1659927123151100, cur_ts_=1659927123117352, delta=33748) [2022-08-08 10:52:03.227157] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:03.227367] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668169, retry interval=1000000, ret=-4179) [2022-08-08 10:52:03.366859] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659927123366849, cur_ts_=1659927123324190, delta=42659) [2022-08-08 10:52:03.424089] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:03.424332] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668174, retry interval=1000000, ret=-4179) [2022-08-08 10:52:03.429259] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:03.435756] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=435] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10193) [2022-08-08 10:52:03.559526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:03.570818] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=38822) [2022-08-08 10:52:03.571307] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=58472, time_dist: generate_cluster_version=58468) [2022-08-08 10:52:03.576936] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:03.578608] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=50576, time_dist: do_cluster_heartbeat=9, generate_cluster_version=50558, cluster_service_self_check=4) [2022-08-08 10:52:03.611504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:03.652689] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25000) [2022-08-08 10:52:03.653489] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26637, time_dist: do_cluster_heartbeat=11, generate_cluster_version=26623) [2022-08-08 10:52:03.655534] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18337, time_dist: do_cluster_heartbeat=88, generate_cluster_version=18245) [2022-08-08 10:52:03.722170] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:03.766388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:03.778357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:03.778458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:03.783096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:03.783162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:03.788892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:03.830490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:03.892814] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:03.921032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:03.927948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:03.928049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:03.938428] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=434] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:03.939874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:03.951253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:03.967611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:04.010928] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.015224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.038309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:04.039196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.046951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:04.047020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.049551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.053445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:04.054483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:04.059920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:04.060448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:04.061376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:04.068096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.070786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.081147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:04.103025] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:04.106283] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:04.106346] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:04.119359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:04.131443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.136742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.147712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.168478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:04.183510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=207] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.186020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:04.194877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:04.196317] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:04.214630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.229513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A2-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:04.231554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:04.251534] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:04.251597] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] task process failed, start retry(max retry time=9223372036854668168, retry interval=1000000, ret=-4179) [2022-08-08 10:52:04.301926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.318912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:04.320815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:04.371979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.401759] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:04.403296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.414553] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.446555] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:04.446593] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668173, retry interval=1000000, ret=-4179) [2022-08-08 10:52:04.470649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:04.479119] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.553026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:04.573287] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:04.667061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:04.835807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:05.109470] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:05.112648] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:05.112672] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:05.275480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:05.277162] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10433, time_dist: generate_cluster_version=10430) [2022-08-08 10:52:05.278312] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:05.278375] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668167, retry interval=1000000, ret=-4179) [2022-08-08 10:52:05.476297] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15669) [2022-08-08 10:52:05.477653] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:05.477705] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668172, retry interval=1000000, ret=-4179) [2022-08-08 10:52:05.480240] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21424, time_dist: do_cluster_heartbeat=11, generate_cluster_version=21410) [2022-08-08 10:52:06.115418] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:06.120270] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:06.120308] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:06.280733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F2A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.280816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F2A-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.281002] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F2A-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.281144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F2A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.309083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.309171] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.310826] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:06.310875] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668166, retry interval=1000000, ret=-4179) [2022-08-08 10:52:06.480078] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480185] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=95] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480194] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480496] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480536] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480549] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480939] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.480981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48A5-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:06.503115] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:06.503319] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] task process failed, start retry(max retry time=9223372036854668171, retry interval=1000000, ret=-4179) [2022-08-08 10:52:07.122893] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:07.127032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:07.127074] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:07.336886] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:07.337041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:07.338694] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:07.338748] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668165, retry interval=1000000, ret=-4179) [2022-08-08 10:52:07.476361] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927127476354, cur_ts_=1659927127455644, delta=20710) [2022-08-08 10:52:07.540650] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:07.540693] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668170, retry interval=1000000, ret=-4179) [2022-08-08 10:52:07.780501] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18645) [2022-08-08 10:52:08.131683] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:08.134961] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:08.135001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:08.385211] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:08.385231] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:08.386853] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:08.386885] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668164, retry interval=1000000, ret=-4179) [2022-08-08 10:52:08.575788] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927128575783, cur_ts_=1659927128521584, delta=54199) [2022-08-08 10:52:08.597272] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:08.597410] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668169, retry interval=1000000, ret=-4179) [2022-08-08 10:52:08.745976] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927128745968, cur_ts_=1659927128724247, delta=21721) [2022-08-08 10:52:08.763738] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B95-0-0] [lt=143] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.763763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B95-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.781779] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B95-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.781818] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2B95-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.782093] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.782143] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.782161] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:08.782174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:09.137607] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:09.140610] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:09.140636] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:09.418607] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:09.419142] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:09.419192] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668163, retry interval=1000000, ret=-4179) [2022-08-08 10:52:09.621304] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927129621296, cur_ts_=1659927129585000, delta=36296) [2022-08-08 10:52:09.653363] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:52:09.653377] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:52:09.667826] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:09.667891] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668168, retry interval=1000000, ret=-4179) [2022-08-08 10:52:09.779008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:09.790676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:09.928897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:09.931257] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14645760}) [2022-08-08 10:52:10.079335] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14978, time_dist: do_cluster_heartbeat=11, generate_cluster_version=14962) [2022-08-08 10:52:10.107011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:10.111246] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927128249296, end_time=1659927130111233, elapsed_time=1861937, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:10.142480] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:10.147282] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:10.147324] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:10.149590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:10.173245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.195720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=3331] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:10.196101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=2860] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.196523] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:10.216555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:10.250570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.259852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=49247] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.260011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=3003] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:10.322275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:10.325593] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31535) [2022-08-08 10:52:10.325772] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32468, time_dist: generate_cluster_version=32466) [2022-08-08 10:52:10.326498] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927130326493, cur_ts_=1659927130294468, delta=32025) [2022-08-08 10:52:10.324398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.439121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:10.449127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:10.449230] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:10.450813] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:10.450875] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668162, retry interval=1000000, ret=-4179) [2022-08-08 10:52:10.484227] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.626393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:10.667203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:10.671900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:10.696155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:10.696189] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:10.697599] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:10.697653] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668167, retry interval=1000000, ret=-4179) [2022-08-08 10:52:10.697768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=235] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:10.814432] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:10.814529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:10.814702] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:10.814880] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:11.137043] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10705, time_dist: do_cluster_heartbeat=9, generate_cluster_version=10692) [2022-08-08 10:52:11.152317] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:11.155467] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:11.155509] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:11.476557] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:11.476716] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] task process failed, start retry(max retry time=9223372036854668161, retry interval=1000000, ret=-4179) [2022-08-08 10:52:11.705203] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:11.726147] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:11.726209] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668166, retry interval=1000000, ret=-4179) [2022-08-08 10:52:11.940591] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] [trans sampling] (TRACE=begin_ts=1659927131939590 2022-08-08 02:52:11.939590|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:6305968121161365614, inc:84075565, addr:{ip:"192.168.11.128", port:2882}, t:1659927131939000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5A77-0-0, start_time:1659927131939466, snapshot:1659927131854220|[end_task] u=127 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927131939736|[calc_checksum_by_commit] u=20 ret:0, checksum:1025627104, checksum_log_ts:9223372036854775807|[submit_log] u=70 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=687 trans_version:1659927131939759, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299477, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=962) [2022-08-08 10:52:12.157931] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:12.164326] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:12.164368] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:12.501111] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:12.501169] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:12.502242] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:12.502307] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668160, retry interval=1000000, ret=-4179) [2022-08-08 10:52:12.627522] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:12.750586] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:12.750606] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:12.752398] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:12.752454] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668165, retry interval=1000000, ret=-4179) [2022-08-08 10:52:12.823139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BA1-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:12.823188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BA1-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:12.823195] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BA1-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:12.823199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BA1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:13.166596] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:13.169674] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:13.169718] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:13.356875] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10212, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10200) [2022-08-08 10:52:13.452970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=202] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:13.511679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.511700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.511765] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.511922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.511987] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512125] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512291] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512358] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512672] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512788] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.512932] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.513050] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.513189] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.513308] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.513440] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.513855] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.513880] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C472E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.541184] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:13.541223] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668159, retry interval=1000000, ret=-4179) [2022-08-08 10:52:13.583130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=2454] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:13.601844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:13.634922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:13.782781] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.782875] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:13.784168] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:13.784229] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668164, retry interval=1000000, ret=-4179) [2022-08-08 10:52:13.787341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:13.795368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:13.803279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:13.805542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:13.805607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:13.813524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:13.845606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:13.910431] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927133910424, cur_ts_=1659927133882307, delta=28117) [2022-08-08 10:52:13.910962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:13.935246] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24399) [2022-08-08 10:52:13.935648] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25531, time_dist: generate_cluster_version=25529) [2022-08-08 10:52:13.936211] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25127, time_dist: do_cluster_heartbeat=24, generate_cluster_version=25100) [2022-08-08 10:52:13.944631] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:13.945326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:13.945501] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:13.958007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:13.958191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:13.968166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:13.988981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:14.028237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.041564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.060578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.061020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:14.064356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:14.065036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.071638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:14.074908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.077272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:14.082742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:14.082957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:14.083040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:14.085805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.092923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.103219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:14.141273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:14.147714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.158215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.167119] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.171987] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:14.178415] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:14.178439] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:14.188510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:14.200279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.210919] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:14.211290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:14.219747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:14.230095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.328846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.334172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:14.339669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:14.396058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.426297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.427503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:14.430137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.493684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:14.499881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.564403] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:14.564426] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:14.566919] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=196] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:14.566984] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668158, retry interval=1000000, ret=-4179) [2022-08-08 10:52:14.572239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:14.648786] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:14.685362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:14.808899] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:14.808923] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:14.810629] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:14.810692] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668163, retry interval=1000000, ret=-4179) [2022-08-08 10:52:14.841892] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BA7-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:14.842029] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BA7-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:14.843632] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:14.851574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=370] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:15.181835] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:15.185225] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:15.185268] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:15.246195] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:52:15.572354] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4771-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:15.572415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4771-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:15.594227] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:15.594261] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668157, retry interval=1000000, ret=-4179) [2022-08-08 10:52:15.838046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:15.840243] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:15.840306] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668162, retry interval=1000000, ret=-4179) [2022-08-08 10:52:16.187733] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:16.191389] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:16.191434] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:16.621174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:16.621221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:16.622276] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:16.622470] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668156, retry interval=1000000, ret=-4179) [2022-08-08 10:52:16.863985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:16.864123] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:16.865401] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:16.865484] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] task process failed, start retry(max retry time=9223372036854668161, retry interval=1000000, ret=-4179) [2022-08-08 10:52:17.195285] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:17.198442] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:17.198489] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:17.647131] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:17.647154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:17.648733] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:17.648770] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668155, retry interval=1000000, ret=-4179) [2022-08-08 10:52:17.860857] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927137860850, cur_ts_=1659927137831244, delta=29606) [2022-08-08 10:52:17.874532] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4774-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:17.874566] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4774-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:17.895981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:17.898157] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=2147] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:17.899500] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:17.899550] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668160, retry interval=1000000, ret=-4179) [2022-08-08 10:52:18.200856] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:18.205051] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:18.205108] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:18.655685] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.655892] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.656022] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.656200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.656372] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.656518] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.656737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.656848] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.657013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.657045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F3C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.682770] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:18.682842] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668154, retry interval=1000000, ret=-4179) [2022-08-08 10:52:18.777086] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12805) [2022-08-08 10:52:18.777331] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14226, time_dist: generate_cluster_version=14224) [2022-08-08 10:52:18.811977] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33188) [2022-08-08 10:52:18.812192] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34797, time_dist: print_stat=4, do_cluster_heartbeat=3, generate_cluster_version=34788) [2022-08-08 10:52:18.812251] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927138812245, cur_ts_=1659927138786091, delta=26154) [2022-08-08 10:52:18.886068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB3-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:18.886142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB3-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:18.886235] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB3-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:18.890205] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:18.901419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48BA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.901487] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48BA-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.928571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.928611] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:18.932976] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=3473] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:18.933717] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=621] [dc=0] task process failed, start retry(max retry time=9223372036854668159, retry interval=1000000, ret=-4179) [2022-08-08 10:52:18.946306] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11180) [2022-08-08 10:52:18.964723] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17469) [2022-08-08 10:52:18.964964] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18433, time_dist: generate_cluster_version=18430) [2022-08-08 10:52:18.988250] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10181, time_dist: generate_cluster_version=10177) [2022-08-08 10:52:19.210715] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:19.214767] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:19.214793] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:19.439895] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26808, time_dist: generate_cluster_version=26805) [2022-08-08 10:52:19.650126] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:52:19.650156] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:52:19.718134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:19.719609] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:19.719671] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668153, retry interval=1000000, ret=-4179) [2022-08-08 10:52:19.798309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=922] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:19.798609] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26236) [2022-08-08 10:52:19.798989] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29298, time_dist: generate_cluster_version=29205, cluster_service_self_check=90) [2022-08-08 10:52:19.814891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:19.951839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:19.961606] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:19.961654] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668158, retry interval=1000000, ret=-4179) [2022-08-08 10:52:19.967752] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927139967744, cur_ts_=1659927139940697, delta=27047) [2022-08-08 10:52:19.967805] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27714) [2022-08-08 10:52:19.968322] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29568, time_dist: do_cluster_heartbeat=9, generate_cluster_version=29556) [2022-08-08 10:52:19.975725] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12305, time_dist: generate_cluster_version=12301) [2022-08-08 10:52:19.997254] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14650144}) [2022-08-08 10:52:20.133215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:20.167348] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:20.187007] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=163] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927138249669, end_time=1659927140186994, elapsed_time=1937325, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:20.198320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.216115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.216678] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:20.219167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=189] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:20.219750] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:20.219791] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:20.220576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:20.241254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:20.271229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.281579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.281690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:20.340610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=3814] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:20.341112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=2477] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.454913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:20.501456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.646457] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:20.687826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:20.721470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:20.722334] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48BD-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:20.740101] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:20.740119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:20.743429] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:20.743482] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668152, retry interval=1000000, ret=-4179) [2022-08-08 10:52:20.794286] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927140794277, cur_ts_=1659927140759280, delta=34997) [2022-08-08 10:52:20.908404] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB9-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:20.908537] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB9-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:20.908636] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB9-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:20.908686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BB9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:20.982009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:20.983212] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:20.983280] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668157, retry interval=1000000, ret=-4179) [2022-08-08 10:52:21.223273] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:21.226605] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:21.226648] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:21.746793] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C473A-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:21.746878] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C473A-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:21.768119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:21.771507] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:21.771576] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668151, retry interval=1000000, ret=-4179) [2022-08-08 10:52:22.006388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:22.006409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:22.007971] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:22.008007] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668156, retry interval=1000000, ret=-4179) [2022-08-08 10:52:22.228973] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:22.232271] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:22.232341] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:22.794077] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:22.794116] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668150, retry interval=1000000, ret=-4179) [2022-08-08 10:52:22.924478] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BBF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:22.924495] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BBF-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:22.924562] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BBF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:22.924638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BBF-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:23.026489] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:23.026573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:23.027764] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:23.027833] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668155, retry interval=1000000, ret=-4179) [2022-08-08 10:52:23.234787] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:23.237979] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:23.238029] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:23.470374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:23.600241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:23.622369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:23.653574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:23.802767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:23.815937] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:23.815957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:23.816057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:23.819590] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:23.819660] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668149, retry interval=1000000, ret=-4179) [2022-08-08 10:52:23.823632] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:23.823905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:23.826490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:23.834295] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:23.866315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:23.927893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:23.963502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:23.970608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:23.970703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:23.980667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:23.981322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:23.992328] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:24.007548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:24.032981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.033015] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.050421] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.050450] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.050844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=2989] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.053546] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:24.053667] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668154, retry interval=1000000, ret=-4179) [2022-08-08 10:52:24.056925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.077709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.078603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:24.083447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.083446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=635] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:24.092432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:24.099526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:24.099633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:24.100764] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:24.101976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.102796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:24.109845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.117668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.120140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:24.160781] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:24.171638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.173744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.192040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.207419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:24.223535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.228364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:24.232269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:24.240789] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:24.241003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:24.245327] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:24.245368] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:24.249589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.344084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.356282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:24.357984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:24.387728] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927144387721, cur_ts_=1659927144362757, delta=24964) [2022-08-08 10:52:24.418880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.446002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.448571] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=1093] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.457462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:24.510276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:24.520664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.592962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:24.708253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.715686] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=499] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:24.822138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48C3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.822211] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48C3-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.822351] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48C3-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.822562] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48C3-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:24.842148] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:24.842209] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668148, retry interval=1000000, ret=-4179) [2022-08-08 10:52:24.873149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:24.940327] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BC5-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:24.940380] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BC5-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:24.940388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BC5-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:24.945288] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:24.945360] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:24.945401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:25.056443] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F45-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.056546] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F45-0-0] [lt=58] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.056733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F45-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.056821] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F45-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.058874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.058967] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.077127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.077255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:25.078355] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:25.078391] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668153, retry interval=1000000, ret=-4179) [2022-08-08 10:52:25.248077] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:25.251296] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:25.251333] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:25.554858] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19914) [2022-08-08 10:52:25.555142] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20931, time_dist: do_cluster_heartbeat=9, generate_cluster_version=20920) [2022-08-08 10:52:25.871130] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:25.871163] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668147, retry interval=1000000, ret=-4179) [2022-08-08 10:52:25.949277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:25.949342] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:26.086608] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.086638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.086802] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.086986] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.087113] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.087298] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.087544] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.087799] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.087921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.088055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.088150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4780-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.108303] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.108414] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.110499] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:26.110648] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668152, retry interval=1000000, ret=-4179) [2022-08-08 10:52:26.256012] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:26.261614] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:26.261667] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:26.873513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F48-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.898353] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.898373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:26.899997] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:26.900029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668146, retry interval=1000000, ret=-4179) [2022-08-08 10:52:26.953401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BCB-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:26.953419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BCB-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:26.953424] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BCB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:27.133833] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:27.133899] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:27.138786] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:27.138918] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668151, retry interval=1000000, ret=-4179) [2022-08-08 10:52:27.141162] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] [trans sampling] (TRACE=begin_ts=1659927147139463 2022-08-08 02:52:27.139463|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:15704188990590520447, inc:84076147, addr:{ip:"192.168.11.128", port:2882}, t:1659927147139000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=25 ret:0, left_time:10000000, uref:1073741825|[start_task] u=10 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7A91-0-0, start_time:1659927147139304, snapshot:1659927147070920|[end_task] u=131 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=12 stc:1659927147139636|[calc_checksum_by_commit] u=20 ret:0, checksum:1140854932, checksum_log_ts:9223372036854775807|[submit_log] u=73 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1332 trans_version:1659927147139193, uref:1073741825|[exiting] u=23 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987226, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=1651) [2022-08-08 10:52:27.265506] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:27.272093] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:27.272158] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:27.930707] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:27.931329] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=512] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:27.965544] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2BCE-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927147963842 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927147965119 abs_expired_time=1659927149894880 stmt_timeout=1929761 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21587772 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927147873028 start_version=-1 trans_version=1659927147963832 commit_version=0 stmt_start_time=1659927147958068 abs_expired_time=1659927177858405 stmt_timeout=29900337 abs_lock_wait_timeout=1659927177858405 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:10521279411652291806, inc:84076176, addr:"192.168.11.128:2882", t:1659927147958068} checksum_log_ts=9223372036854775807, version=1659927147963842) [2022-08-08 10:52:27.969597] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:27.969655] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668145, retry interval=1000000, ret=-4179) [2022-08-08 10:52:28.160593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:28.160864] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=140] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:28.166004] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:28.166047] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668150, retry interval=1000000, ret=-4179) [2022-08-08 10:52:28.275280] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:28.280788] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:28.280815] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:28.654879] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927148654872, cur_ts_=1659927148628131, delta=26741) [2022-08-08 10:52:28.987980] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:28.988001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:29.016743] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:29.021769] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927149021757, cur_ts_=1659927149000751, delta=21006) [2022-08-08 10:52:29.031510] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=262] [dc=0] task process failed, start retry(max retry time=9223372036854668144, retry interval=1000000, ret=-4179) [2022-08-08 10:52:29.033159] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32899) [2022-08-08 10:52:29.034356] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35647, time_dist: do_cluster_heartbeat=11, generate_cluster_version=35632) [2022-08-08 10:52:29.212439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:29.212566] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:29.219056] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:29.219113] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668149, retry interval=1000000, ret=-4179) [2022-08-08 10:52:29.267168] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10420) [2022-08-08 10:52:29.267467] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11866, time_dist: generate_cluster_version=11863) [2022-08-08 10:52:29.285236] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:29.292026] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:29.292052] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:29.627392] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11052, time_dist: generate_cluster_version=11048) [2022-08-08 10:52:29.687162] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10933) [2022-08-08 10:52:29.687406] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14523, time_dist: print_stat=6, generate_cluster_version=14515) [2022-08-08 10:52:29.690306] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:52:29.691778] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=982] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:52:29.760671] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927149760666, cur_ts_=1659927149738534, delta=22132) [2022-08-08 10:52:29.816378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:29.832713] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16179, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16163) [2022-08-08 10:52:29.834367] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16906, time_dist: generate_cluster_version=16901) [2022-08-08 10:52:29.854200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:29.986894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:29.987333] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] clock out of order(cur_ts=1659927149987329, cur_ts_=1659927149957746, delta=29583) [2022-08-08 10:52:30.092021] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=41675) [2022-08-08 10:52:30.096187] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=72631, time_dist: do_cluster_heartbeat=11, generate_cluster_version=72615) [2022-08-08 10:52:30.106683] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14617) [2022-08-08 10:52:30.107054] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=81266, time_dist: generate_cluster_version=81263) [2022-08-08 10:52:30.140342] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:30.141883] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:30.143466] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] task process failed, start retry(max retry time=9223372036854668143, retry interval=1000000, ret=-4179) [2022-08-08 10:52:30.149993] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=7] [dc=0] [trans sampling] (TRACE=begin_ts=1659927150142887 2022-08-08 02:52:30.142887|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:8087008807223691101, inc:84076257, addr:{ip:"192.168.11.128", port:2882}, t:1659927150142000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=38 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7A96-0-0, start_time:1659927150142749, snapshot:1659927150110819|[end_task] u=117 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927150143055|[calc_checksum_by_commit] u=19 ret:0, checksum:328033512, checksum_log_ts:9223372036854775807|[submit_log] u=31 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=6780 trans_version:1659927150142495, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:987231, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=15 arg1:140085233102640|total_timeu=7051) [2022-08-08 10:52:30.158369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:30.183749] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:30.215210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.221294] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4786-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:30.221622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4786-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:30.247354] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:30.249804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:30.250286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.259700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:30.260803] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:30.264825] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:30.264892] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668148, retry interval=1000000, ret=-4179) [2022-08-08 10:52:30.288826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.295238] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:30.298473] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:30.298517] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:30.321708] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927150321699, cur_ts_=1659927150297731, delta=23968) [2022-08-08 10:52:30.325519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:30.325811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.362195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:30.362309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.363522] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14654592}) [2022-08-08 10:52:30.472170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:30.523142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.558338] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927148250972, end_time=1659927150558326, elapsed_time=2307354, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:30.667794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:30.703614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:30.745256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:31.113802] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BD7-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:31.113869] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BD7-0-0] [lt=55] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:31.113879] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BD7-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:31.113884] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BD7-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:31.165439] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:31.165508] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668142, retry interval=1000000, ret=-4179) [2022-08-08 10:52:31.289572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:31.290245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:31.294145] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:31.294184] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668147, retry interval=1000000, ret=-4179) [2022-08-08 10:52:31.326704] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:31.330507] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:31.330551] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:31.719503] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927151719494, cur_ts_=1659927151686640, delta=32854) [2022-08-08 10:52:32.135065] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:32.135126] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:32.140103] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927152129545 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927152127577 abs_expired_time=1659927154059116 stmt_timeout=1931539 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21587828 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927152040190 start_version=-1 trans_version=0 commit_version=0 stmt_start_time=1659927152127577 abs_expired_time=1659927182027887 stmt_timeout=29900310 abs_lock_wait_timeout=1659927182027887 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:2836971295773674976, inc:84076329, addr:"192.168.11.128:2882", t:1659927152127577} checksum_log_ts=9223372036854775807, version=1659927152129545) [2022-08-08 10:52:32.186620] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.190134] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:32.190191] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668141, retry interval=1000000, ret=-4179) [2022-08-08 10:52:32.295982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.296204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.296502] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.296763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.297193] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.297596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.297617] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.297859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4749-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.298467] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.315564] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:32.317315] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:32.317352] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668146, retry interval=1000000, ret=-4179) [2022-08-08 10:52:32.338069] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:32.343500] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=157] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:32.343526] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:32.482691] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1532] [dc=0] clock out of order(cur_ts=1659927152482682, cur_ts_=1659927152440769, delta=41913) [2022-08-08 10:52:32.627890] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:33.146983] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BDD-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:33.147239] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BDD-0-0] [lt=238] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:33.147254] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BDD-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:33.147260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BDD-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:33.221758] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:33.221905] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=135] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:33.226895] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:33.227246] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668140, retry interval=1000000, ret=-4179) [2022-08-08 10:52:33.338193] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:33.338228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:33.342707] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:33.342765] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668145, retry interval=1000000, ret=-4179) [2022-08-08 10:52:33.351384] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:33.355878] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:33.355910] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:33.487787] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:33.623121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:33.642542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:33.673976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:33.821082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:33.832945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:33.842074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:33.843026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:33.845369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:33.853384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:33.882663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:33.946634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:33.981212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:33.987901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:33.990343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:33.998551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:34.000645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.010782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:34.031422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:34.067926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.080010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.094622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.099326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:34.105016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:34.105111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.115670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:34.117254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:34.121821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:34.124791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:34.124940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.125691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:34.135779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.141489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:34.142576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.182349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:34.191280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=275] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.222046] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=266] [dc=0] clock out of order(cur_ts=1659927154221981, cur_ts_=1659927154193734, delta=28247) [2022-08-08 10:52:34.194661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.230967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.233961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:34.261999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:34.263179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:34.263326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:34.264173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.266478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.293830] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:34.293850] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:34.314263] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=299] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:34.314308] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668139, retry interval=1000000, ret=-4179) [2022-08-08 10:52:34.315397] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=51601) [2022-08-08 10:52:34.332064] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=68993, time_dist: generate_cluster_version=68982, cluster_service_self_check=7) [2022-08-08 10:52:34.335086] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19642) [2022-08-08 10:52:34.338008] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=93410, time_dist: generate_cluster_version=93406) [2022-08-08 10:52:34.360271] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:34.366814] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:34.366839] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:34.371369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.374029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:34.377376] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:34.377432] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668144, retry interval=1000000, ret=-4179) [2022-08-08 10:52:34.382957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:34.441069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.462579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.479785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=8491] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.479889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:34.532962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:34.540430] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.611334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:34.732819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:34.773086] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:34.891068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:35.189405] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BE3-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:35.189424] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BE3-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:35.191929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:35.191969] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:35.191998] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:35.192035] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:35.270041] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:52:35.336213] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:35.336356] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668138, retry interval=1000000, ret=-4179) [2022-08-08 10:52:35.374645] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:35.378362] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:35.378428] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:35.379702] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C48D2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:35.402677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:35.402700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:35.404241] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:35.404404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668143, retry interval=1000000, ret=-4179) [2022-08-08 10:52:36.204168] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BE6-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:36.204268] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BE6-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:36.204397] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BE6-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:36.204448] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BE6-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:36.208549] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:36.276296] TRACE [TRACE]ob_trans_define.cpp:1186 [11024][0][YB42C0A80B80-0005E3CF9BC835E8-0-0] [lt=17] [dc=0] [trans error] (TRACE=begin_ts=1659926903642154 2022-08-08 02:48:23.642154|[start_trans] u=0 ret:0, trans_id:{hash:14122346763008492130, inc:84066165, addr:{ip:"192.168.11.128", port:2882}, t:1659926903641018}|[start_stmt_v2] u=13 ret:0, sql_no:4294967297|[start_participant] u=32 ret:0|[get_store_ctx] u=98136 ret:0|[end_participant] u=169 ret:0|[end_stmt_v2] u=4 ret:0|[start_stmt_v2] u=5898287 ret:0, sql_no:8589934594|[start_participant] u=31 ret:0|[get_store_ctx] u=32 ret:0|[end_participant] u=17151 ret:0|[end_stmt_v2] u=5 ret:0|[start_stmt_v2] u=11992987 ret:0, sql_no:12884901891|[start_participant] u=25 ret:0|[get_store_ctx] u=29 ret:0|[end_participant] u=139 ret:0|[end_stmt_v2] u=3 ret:0|[start_stmt_v2] u=10642081 ret:0, sql_no:17179869188|[start_participant] u=18 ret:0|[get_store_ctx] u=114 ret:0|[get_store_ctx] u=107 ret:0|[get_store_ctx] u=23 ret:0|[end_participant] u=37 ret:0|[end_stmt_v2] u=2 ret:0|[start_stmt_v2] u=3343791 ret:0, sql_no:21474836485|[start_participant] u=23 ret:0|[get_store_ctx] u=71 ret:0|[get_store_ctx] u=1302 ret:0|[get_store_ctx] u=41 ret:0|[get_store_ctx] u=25 ret:0|[get_store_ctx] u=86 ret:0|[end_participant] u=85 ret:0|[end_stmt_v2] u=6 ret:0|[start_stmt_v2] u=51376471 ret:0, sql_no:25769803782|[start_participant] u=18 ret:0|[get_store_ctx] u=17 ret:0|[end_participant] u=49 ret:0|[end_stmt_v2] u=2 ret:0|[start_stmt_v2] u=27533510 ret:-6210, sql_no:25769803782|[start_stmt_v2] u=14026740 ret:-6210, sql_no:25769803782|[end_trans] u=81950292 ret:-6210|total_timeu=206881954) [2022-08-08 10:52:36.372106] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927156372097, cur_ts_=1659927156336324, delta=35773) [2022-08-08 10:52:36.385483] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:36.389971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:36.389995] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:36.405497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:36.409318] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:36.409371] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668137, retry interval=1000000, ret=-4179) [2022-08-08 10:52:36.430640] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:36.430679] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668142, retry interval=1000000, ret=-4179) [2022-08-08 10:52:37.393063] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:37.398866] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:37.398923] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:37.431294] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:37.431338] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:37.436713] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:37.436752] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668136, retry interval=1000000, ret=-4179) [2022-08-08 10:52:37.460412] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:37.462390] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:37.462571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668141, retry interval=1000000, ret=-4179) [2022-08-08 10:52:38.226154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BEC-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:38.226296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BEC-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:38.227856] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:38.227990] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:38.228217] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:38.228365] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:38.263723] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12667, time_dist: generate_cluster_version=12663) [2022-08-08 10:52:38.401927] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:38.408563] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:38.408593] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:38.476828] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16584) [2022-08-08 10:52:38.477646] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18143, time_dist: generate_cluster_version=18141) [2022-08-08 10:52:38.495232] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:38.495280] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668140, retry interval=1000000, ret=-4179) [2022-08-08 10:52:38.495501] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18637) [2022-08-08 10:52:38.496055] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:38.496079] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668135, retry interval=1000000, ret=-4179) [2022-08-08 10:52:38.496855] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22397, time_dist: generate_cluster_version=22395) [2022-08-08 10:52:38.665784] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15191) [2022-08-08 10:52:38.666062] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16290, time_dist: do_cluster_heartbeat=9, generate_cluster_version=16279) [2022-08-08 10:52:38.904600] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21007) [2022-08-08 10:52:38.904924] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22626, time_dist: do_cluster_heartbeat=9, generate_cluster_version=22613) [2022-08-08 10:52:38.906015] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22975, time_dist: generate_cluster_version=22971) [2022-08-08 10:52:39.239462] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BEF-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:39.239480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BEF-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:39.239485] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BEF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:39.239490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BEF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:39.290007] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17691) [2022-08-08 10:52:39.291434] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20620, time_dist: generate_cluster_version=20615) [2022-08-08 10:52:39.327191] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=272] [dc=0] clock out of order(cur_ts=1659927159327182, cur_ts_=1659927159291478, delta=35704) [2022-08-08 10:52:39.415825] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:39.419313] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:39.419342] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:39.525142] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927159525131, cur_ts_=1659927159497792, delta=27339) [2022-08-08 10:52:39.538720] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13159, time_dist: do_cluster_heartbeat=12, generate_cluster_version=13143) [2022-08-08 10:52:39.588869] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:39.588907] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668139, retry interval=1000000, ret=-4179) [2022-08-08 10:52:39.589532] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:39.589556] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668134, retry interval=1000000, ret=-4179) [2022-08-08 10:52:39.650740] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:52:39.650776] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:52:39.843810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:39.870724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=573] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.002046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.094134] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14658880}) [2022-08-08 10:52:40.094254] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17412, time_dist: generate_cluster_version=17409) [2022-08-08 10:52:40.180048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:40.206740] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:40.242399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.272961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.275310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.277152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.278257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=2731] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:40.279436] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17793) [2022-08-08 10:52:40.279899] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19220, time_dist: do_cluster_heartbeat=22, generate_cluster_version=19196) [2022-08-08 10:52:40.297240] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35883, time_dist: generate_cluster_version=35877) [2022-08-08 10:52:40.307258] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20676) [2022-08-08 10:52:40.307698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.349149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.349563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.349930] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=40409) [2022-08-08 10:52:40.380866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.381476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:40.399211] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927158252122, end_time=1659927160399196, elapsed_time=2147074, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:40.421306] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:40.425104] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:40.425135] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:40.493132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.545423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.605423] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:40.606004] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:40.606044] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:40.629814] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:40.631540] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:40.631650] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668133, retry interval=1000000, ret=-4179) [2022-08-08 10:52:40.633534] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=505] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:40.633575] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668138, retry interval=1000000, ret=-4179) [2022-08-08 10:52:40.691131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:40.722275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:40.761954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:41.298880] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BF5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:41.299108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BF5-0-0] [lt=201] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:41.299119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BF5-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:41.299207] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BF5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:41.430011] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:41.433181] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:41.433206] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:41.528908] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10020, time_dist: generate_cluster_version=10015) [2022-08-08 10:52:41.659208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:41.659346] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:41.660860] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:41.660916] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668132, retry interval=1000000, ret=-4179) [2022-08-08 10:52:41.665619] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:41.665662] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668137, retry interval=1000000, ret=-4179) [2022-08-08 10:52:42.084751] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927162084742, cur_ts_=1659927162045871, delta=38871) [2022-08-08 10:52:42.189813] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927162189805, cur_ts_=1659927162121484, delta=68321) [2022-08-08 10:52:42.201871] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17994) [2022-08-08 10:52:42.202094] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21071, time_dist: do_cluster_heartbeat=10, generate_cluster_version=21058) [2022-08-08 10:52:42.202680] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12758, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12747) [2022-08-08 10:52:42.437266] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:42.442524] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:42.442573] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:42.674088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.674251] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.674443] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.674578] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.676242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.676386] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.676402] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.676637] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=222] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.676650] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.676658] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4758-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.698739] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:42.698981] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668131, retry interval=1000000, ret=-4179) [2022-08-08 10:52:42.699034] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10287, time_dist: do_cluster_heartbeat=11, generate_cluster_version=10274) [2022-08-08 10:52:42.700351] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:42.701903] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:42.701935] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668136, retry interval=1000000, ret=-4179) [2022-08-08 10:52:43.322045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:43.322085] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFB-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:43.322314] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFB-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:43.322345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFB-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:43.448242] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:43.451323] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:43.451348] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:43.505402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:43.640219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:43.660529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:43.690808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:43.701711] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F60-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:43.701817] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F60-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:43.701857] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F60-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:43.739773] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:43.739811] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668135, retry interval=1000000, ret=-4179) [2022-08-08 10:52:43.742725] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:43.742787] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668130, retry interval=1000000, ret=-4179) [2022-08-08 10:52:43.860547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:43.860597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:43.860665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:43.861212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:43.862786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:43.878953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:43.879247] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16557) [2022-08-08 10:52:43.881585] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=384] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20428, time_dist: do_cluster_heartbeat=10, generate_cluster_version=20414) [2022-08-08 10:52:43.896429] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16426) [2022-08-08 10:52:43.899671] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21748, time_dist: do_cluster_heartbeat=8, generate_cluster_version=21736) [2022-08-08 10:52:43.903593] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:43.983183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.014714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:44.016537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:44.016866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.016914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.025380] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.028130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:44.055927] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:44.087214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=157] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.099386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.119708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=233] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.120226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:44.120975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:44.125312] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.132835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:44.140762] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:44.141212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:44.141291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.141757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:44.149603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:44.151682] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.161503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.163859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:44.198257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=1253] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:44.215815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.217891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=30862] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.250282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.255509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:44.278303] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:44.282705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:44.285492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.288662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.288719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:44.328984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.329002] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFE-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.329049] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.329064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2BFE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.330700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.330726] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.330745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:44.391677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.392679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:44.399090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:44.453781] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:44.457006] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:44.457044] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:44.462591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.486899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.499816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.503665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:44.556899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:44.563819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.633282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:44.752061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:44.771261] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:44.771312] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668129, retry interval=1000000, ret=-4179) [2022-08-08 10:52:44.772305] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:44.772323] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:44.773543] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:44.773603] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668134, retry interval=1000000, ret=-4179) [2022-08-08 10:52:44.838963] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:44.912332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:45.302686] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] [trans sampling] (TRACE=begin_ts=1659927165301504 2022-08-08 02:52:45.301504|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:13596589388232901679, inc:84076831, addr:{ip:"192.168.11.128", port:2882}, t:1659927165298000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AB8EC-0-0, start_time:1659927165301387, snapshot:1659927165297901|[end_task] u=188 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927165301709|[calc_checksum_by_commit] u=20 ret:0, checksum:650509325, checksum_log_ts:9223372036854775807|[submit_log] u=285 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=595 trans_version:1659927165298912, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8314237, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233538864|total_timeu=1142) [2022-08-08 10:52:45.337055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C01-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:45.337071] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C01-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:45.342299] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:45.459652] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:45.464191] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:45.464230] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:45.800129] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:45.800165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668128, retry interval=1000000, ret=-4179) [2022-08-08 10:52:45.805215] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:45.805253] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668133, retry interval=1000000, ret=-4179) [2022-08-08 10:52:46.350623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C04-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:46.350643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C04-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:46.350648] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C04-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:46.350661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C04-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:46.350855] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C04-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:46.351210] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C04-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:46.466990] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:46.470102] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:46.470139] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:46.807068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C479E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:46.807117] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C479E-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:46.827666] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:46.827719] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668127, retry interval=1000000, ret=-4179) [2022-08-08 10:52:46.844880] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:46.844978] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668132, retry interval=1000000, ret=-4179) [2022-08-08 10:52:47.360504] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:47.360579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:47.472446] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:47.477651] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:47.477695] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:47.853886] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:47.853922] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668126, retry interval=1000000, ret=-4179) [2022-08-08 10:52:47.865479] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:47.865501] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:47.868236] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:47.868413] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668131, retry interval=1000000, ret=-4179) [2022-08-08 10:52:48.368727] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] [trans sampling] (TRACE=begin_ts=1659927168361733 2022-08-08 02:52:48.361733|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:5699905654989648273, inc:84076945, addr:{ip:"192.168.11.128", port:2882}, t:1659927168360000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5B2D-0-0, start_time:1659927168361616, snapshot:1659927168319562|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927168361875|[calc_checksum_by_commit] u=20 ret:0, checksum:1523541030, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=6694 trans_version:1659927168361896, uref:1073741825|[exiting] u=26 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299659, logging:0, pending:0, uref:1|[end_trans_cb] u=7 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=6952) [2022-08-08 10:52:48.397905] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10824) [2022-08-08 10:52:48.479614] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:48.494847] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:48.494873] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:48.497883] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10322, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10308) [2022-08-08 10:52:48.744667] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10707) [2022-08-08 10:52:48.744995] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11930, time_dist: generate_cluster_version=11928) [2022-08-08 10:52:48.860394] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:48.860437] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:48.870284] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C47A1-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:48.870359] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C47A1-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:48.889353] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:48.889400] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668125, retry interval=1000000, ret=-4179) [2022-08-08 10:52:48.891830] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:48.891972] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] task process failed, start retry(max retry time=9223372036854668130, retry interval=1000000, ret=-4179) [2022-08-08 10:52:48.959023] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15228) [2022-08-08 10:52:48.959464] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17075, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17062) [2022-08-08 10:52:48.980965] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21905) [2022-08-08 10:52:48.981909] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26920, time_dist: generate_cluster_version=26916) [2022-08-08 10:52:49.095874] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10814) [2022-08-08 10:52:49.096082] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11616, time_dist: do_cluster_heartbeat=7, generate_cluster_version=11607) [2022-08-08 10:52:49.128665] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12114) [2022-08-08 10:52:49.128868] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13065, time_dist: generate_cluster_version=13063) [2022-08-08 10:52:49.375513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C0D-0-0] [lt=136] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:49.378175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:49.378874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=658] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:49.378898] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:49.378918] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:49.497330] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:49.500362] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:49.500433] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:49.649997] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:52:49.650026] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:52:49.853114] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=10293] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14663168}) [2022-08-08 10:52:49.866132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:49.886500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:49.937925] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17509, time_dist: generate_cluster_version=17506) [2022-08-08 10:52:49.951169] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:49.951206] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668129, retry interval=1000000, ret=-4179) [2022-08-08 10:52:49.960489] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:49.960528] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668124, retry interval=1000000, ret=-4179) [2022-08-08 10:52:50.028284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:50.057616] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927168253309, end_time=1659927170057603, elapsed_time=1804294, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:50.201140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:50.230274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:50.261517] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=301] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.292211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=398] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.292361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:50.302521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=775] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:50.302828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:50.323387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.371257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:50.372671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.393679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C10-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.393697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C10-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.393913] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C10-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.393992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C10-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.397180] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.397278] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.397317] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.397336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:50.402326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.402443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:50.502497] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:50.508321] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:50.508664] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=328] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:50.511143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:50.562642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.709398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:50.740758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:50.784032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:50.831603] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] clock out of order(cur_ts=1659927170831594, cur_ts_=1659927170811492, delta=20102) [2022-08-08 10:52:50.983344] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:50.983381] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668128, retry interval=1000000, ret=-4179) [2022-08-08 10:52:50.985356] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:50.985384] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668123, retry interval=1000000, ret=-4179) [2022-08-08 10:52:51.221794] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:52:51.466708] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927171466701, cur_ts_=1659927171428602, delta=38099) [2022-08-08 10:52:51.510866] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:51.514277] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:51.514316] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:51.529205] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10621, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10607) [2022-08-08 10:52:51.970321] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10992) [2022-08-08 10:52:51.970534] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11996, time_dist: generate_cluster_version=11994) [2022-08-08 10:52:52.007705] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:52.007744] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668122, retry interval=1000000, ret=-4179) [2022-08-08 10:52:52.011684] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:52.011773] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668127, retry interval=1000000, ret=-4179) [2022-08-08 10:52:52.415208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C16-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:52.415384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C16-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:52.415411] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C16-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:52.415419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C16-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:52.416431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:52.516735] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:52.520699] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:52.520726] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:52.628363] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:53.013457] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C47A7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:53.017117] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C47A7-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:53.017155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C47A7-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:53.050020] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:53.050061] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668121, retry interval=1000000, ret=-4179) [2022-08-08 10:52:53.057319] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:53.057368] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668126, retry interval=1000000, ret=-4179) [2022-08-08 10:52:53.520882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:53.523113] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:53.526406] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:53.526444] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:53.663540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:53.676730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:53.707614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:53.879931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:53.879982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:53.885380] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:53.885401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:53.885437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:53.895450] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:53.926517] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:53.999958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.031180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:54.034707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.039567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.041677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:54.050338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.052109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:54.076459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:54.082069] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:54.082128] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668120, retry interval=1000000, ret=-4179) [2022-08-08 10:52:54.082999] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:54.083028] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668125, retry interval=1000000, ret=-4179) [2022-08-08 10:52:54.105952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.117140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.137306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=1762] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.143205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=1244] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:54.143204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=510] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:52:54.145891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.156077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:54.160494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.162104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:54.162173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:52:54.163978] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:54.166487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:54.167943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.185767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.188869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:54.218054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:54.235850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.240858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.273189] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:54.274132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.301496] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:54.303451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:54.305308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:52:54.305383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.311865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.414938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:52:54.415485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.417941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:54.431478] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2C1C-0-0] [lt=126] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927174430032 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927174430022 abs_expired_time=1659927176360871 stmt_timeout=1930849 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21588137 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927174382713 start_version=-1 trans_version=1659927174430022 commit_version=0 stmt_start_time=1659927174426898 abs_expired_time=1659927204326956 stmt_timeout=29900058 abs_lock_wait_timeout=1659927204326956 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:11304966532991708180, inc:84077168, addr:"192.168.11.128:2882", t:1659927174426000} checksum_log_ts=9223372036854775807, version=1659927174430032) [2022-08-08 10:52:54.432319] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927174427060 2022-08-08 02:52:54.427060|[init] u=0 arg1:140085233564752, ctx_type:"participant", trans_type:0, trans_id:{hash:11304966532991708180, inc:84077168, addr:{ip:"192.168.11.128", port:2882}, t:1659927174426000}, pkey:{tid:1099511627777, partition_id:0, part_cnt:1}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCAEA102E-0-0, start_time:1659927174426956, snapshot:1659927174382713|[end_task] u=3814 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=14 stc:1659927174430897|[calc_checksum_by_commit] u=28 ret:0, checksum:1275024217, checksum_log_ts:9223372036854775807|[submit_log] u=48 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1251 trans_version:1659927174430022, uref:1073741825|[exiting] u=23 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=4 ret:0, log_type:64, log_id:1655390, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233566128|total_timeu=5218) [2022-08-08 10:52:54.478088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.510198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.519108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.526508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:54.528509] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:54.532690] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:54.532793] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:54.575643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:52:54.583142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.653575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:54.775246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:54.898127] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=531] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:54.927866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:52:55.110799] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:55.110832] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668119, retry interval=1000000, ret=-4179) [2022-08-08 10:52:55.112165] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:55.112188] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668124, retry interval=1000000, ret=-4179) [2022-08-08 10:52:55.294529] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=218] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:52:55.435470] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C1F-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:55.435535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C1F-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:55.435741] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C1F-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:55.435829] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C1F-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:55.535423] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:55.541404] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:55.541431] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:55.932008] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12025) [2022-08-08 10:52:55.932275] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13082, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13069) [2022-08-08 10:52:55.937858] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17870, time_dist: do_cluster_heartbeat=6, generate_cluster_version=17858) [2022-08-08 10:52:56.122085] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:56.122139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:52:56.150511] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:56.150560] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668123, retry interval=1000000, ret=-4179) [2022-08-08 10:52:56.152520] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:56.152579] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668118, retry interval=1000000, ret=-4179) [2022-08-08 10:52:56.255261] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927176255254, cur_ts_=1659927176232367, delta=22887) [2022-08-08 10:52:56.548725] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:56.553281] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:56.553319] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:57.047633] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1971] [dc=0] clock out of order(cur_ts=1659927177047629, cur_ts_=1659927177014078, delta=33551) [2022-08-08 10:52:57.179476] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:57.179517] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668117, retry interval=1000000, ret=-4179) [2022-08-08 10:52:57.182165] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:57.182212] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668122, retry interval=1000000, ret=-4179) [2022-08-08 10:52:57.446343] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C25-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:57.446377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C25-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:57.446501] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C25-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:57.446720] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C25-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:57.447501] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:57.555629] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:57.558976] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:57.559084] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:58.210211] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:58.210249] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668116, retry interval=1000000, ret=-4179) [2022-08-08 10:52:58.213357] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:58.213403] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668121, retry interval=1000000, ret=-4179) [2022-08-08 10:52:58.380973] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14129) [2022-08-08 10:52:58.381160] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16159, time_dist: generate_cluster_version=16158) [2022-08-08 10:52:58.400875] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12712) [2022-08-08 10:52:58.401075] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13674, time_dist: generate_cluster_version=13672) [2022-08-08 10:52:58.471957] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15779) [2022-08-08 10:52:58.561006] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:58.564272] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:58.564308] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:58.958338] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=10088) [2022-08-08 10:52:58.959354] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11804, time_dist: generate_cluster_version=11800) [2022-08-08 10:52:59.261134] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:59.283694] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:52:59.284740] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854668115, retry interval=1000000, ret=-4179) [2022-08-08 10:52:59.284762] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854668120, retry interval=1000000, ret=-4179) [2022-08-08 10:52:59.306000] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] clock out of order(cur_ts=1659927179305991, cur_ts_=1659927179284656, delta=21335) [2022-08-08 10:52:59.306594] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22104) [2022-08-08 10:52:59.399267] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10653, time_dist: generate_cluster_version=10649) [2022-08-08 10:52:59.457775] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2B-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:59.457841] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2B-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:59.457856] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2B-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:59.457923] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:52:59.566685] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:52:59.570361] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:52:59.570388] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:52:59.635518] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31710) [2022-08-08 10:52:59.635791] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32951, time_dist: generate_cluster_version=32948) [2022-08-08 10:52:59.654392] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:52:59.654425] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:52:59.656294] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20703) [2022-08-08 10:52:59.657049] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32660, time_dist: generate_cluster_version=32652, cluster_service_self_check=4) [2022-08-08 10:52:59.809610] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14667552}) [2022-08-08 10:52:59.827346] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31857) [2022-08-08 10:52:59.828169] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34175, time_dist: do_cluster_heartbeat=8, generate_cluster_version=34163) [2022-08-08 10:52:59.884042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:59.907091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:52:59.988731] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927178253472, end_time=1659927179988720, elapsed_time=1735248, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:52:59.988820] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10498) [2022-08-08 10:52:59.989282] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11981, time_dist: print_stat=8, do_cluster_heartbeat=2, generate_cluster_version=11969) [2022-08-08 10:53:00.057797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:00.220280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:00.250952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:00.277828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.293084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:00.293153] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:00.309822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.311447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:00.314466] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:00.314502] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668119, retry interval=1000000, ret=-4179) [2022-08-08 10:53:00.324997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:00.325059] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:00.325109] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668114, retry interval=1000000, ret=-4179) [2022-08-08 10:53:00.325962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:00.344782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.389849] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:00.395088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.420052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.427363] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:00.468427] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:00.468536] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:00.468569] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2E-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:00.468766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:00.468866] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:00.468894] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C2E-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:00.535086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:00.572529] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:00.576085] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:00.576124] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:00.580983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.731106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:00.757377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:00.804680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:01.290839] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927181290829, cur_ts_=1659927181263371, delta=27458) [2022-08-08 10:53:01.371372] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15661) [2022-08-08 10:53:01.384884] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13352) [2022-08-08 10:53:01.385158] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32091, time_dist: do_cluster_heartbeat=11, generate_cluster_version=32077) [2022-08-08 10:53:01.385654] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30751, time_dist: do_cluster_heartbeat=9, generate_cluster_version=30740) [2022-08-08 10:53:01.408081] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:01.408161] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] task process failed, start retry(max retry time=9223372036854668118, retry interval=1000000, ret=-4179) [2022-08-08 10:53:01.417896] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:01.417933] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668113, retry interval=1000000, ret=-4179) [2022-08-08 10:53:01.436589] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16804) [2022-08-08 10:53:01.476398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C31-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:01.476696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C31-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:01.476716] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C31-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:01.476722] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C31-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:01.578320] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:01.581602] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:01.581646] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:02.419799] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F7B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:02.419872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F7B-0-0] [lt=42] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:02.420053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F7B-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:02.420477] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F7B-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:02.445289] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=2616] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:02.446593] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] task process failed, start retry(max retry time=9223372036854668117, retry interval=1000000, ret=-4179) [2022-08-08 10:53:02.448894] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:02.449128] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] task process failed, start retry(max retry time=9223372036854668112, retry interval=1000000, ret=-4179) [2022-08-08 10:53:02.583982] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:02.587102] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:02.587137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:03.472338] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927183472329, cur_ts_=1659927183448736, delta=23593) [2022-08-08 10:53:03.509251] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:03.509313] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668111, retry interval=1000000, ret=-4179) [2022-08-08 10:53:03.510140] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:03.510244] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668116, retry interval=1000000, ret=-4179) [2022-08-08 10:53:03.544385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:03.590874] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:03.594042] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:03.594078] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:03.679311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:03.695153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:03.731273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:03.895771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:03.895846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:03.906386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:03.906487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:03.909397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:03.917016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:03.947567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:04.016604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.047810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:04.051544] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.059107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.062577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:04.069851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=229] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.073037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:04.095388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:04.123291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.142019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.152464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.161765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:04.162735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.168320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:04.177381] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:04.178773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:04.178856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.184089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.184676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:04.185566] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:04.187714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:04.204943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:04.209920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.239214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:04.251573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.256845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.288820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:04.298692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.326481] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:04.327913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:04.328454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:04.329925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.331563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.432131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.435224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:04.436084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:04.496005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.510678] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927184497682 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927184509448 abs_expired_time=1659927186439958 stmt_timeout=1930510 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21588276 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927184496192 start_version=-1 trans_version=1659927184497662 commit_version=0 stmt_start_time=1659927184497662 abs_expired_time=1659927214398619 stmt_timeout=29900957 abs_lock_wait_timeout=1659927214398619 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:9863377570538843880, inc:84077548, addr:"192.168.11.128:2882", t:1659927184497662} checksum_log_ts=9223372036854775807, version=1659927184497682) [2022-08-08 10:53:04.528639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.536601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.545268] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:04.545373] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668115, retry interval=1000000, ret=-4179) [2022-08-08 10:53:04.545799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:04.547696] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:04.547756] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668110, retry interval=1000000, ret=-4179) [2022-08-08 10:53:04.596936] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:04.599043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:04.600966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.603304] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:04.603341] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:04.668793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:04.793579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.950945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:04.956914] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:05.532313] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:05.532467] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:05.532871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:05.532962] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:05.570665] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:05.570955] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] task process failed, start retry(max retry time=9223372036854668114, retry interval=1000000, ret=-4179) [2022-08-08 10:53:05.580656] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=294] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:05.580727] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668109, retry interval=1000000, ret=-4179) [2022-08-08 10:53:05.606957] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:05.610775] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:05.610833] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:06.555723] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927186521005 2022-08-08 02:53:06.521005|[init] u=0 arg1:140085233564752, ctx_type:"participant", trans_type:0, trans_id:{hash:12840817684625282484, inc:84077629, addr:{ip:"192.168.11.128", port:2882}, t:1659927186519000}, pkey:{tid:1099511627777, partition_id:0, part_cnt:1}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCAEA103A-0-0, start_time:1659927186520879, snapshot:1659927186444663|[end_task] u=24401 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=12 stc:1659927186545426|[calc_checksum_by_commit] u=27 ret:0, checksum:1374971016, checksum_log_ts:9223372036854775807|[submit_log] u=31 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=10145 trans_version:1659927186545054, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:1655402, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233566128|total_timeu=34675) [2022-08-08 10:53:06.572839] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.572892] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.572950] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.573051] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.573407] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.573517] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.576759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=3196] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.576775] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.576787] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5F81-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:06.606863] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18094) [2022-08-08 10:53:06.614275] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:06.615591] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27771, time_dist: generate_cluster_version=27768) [2022-08-08 10:53:06.620894] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13832) [2022-08-08 10:53:06.621855] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33020, time_dist: generate_cluster_version=33017) [2022-08-08 10:53:06.623823] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:06.623860] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668113, retry interval=1000000, ret=-4179) [2022-08-08 10:53:06.625046] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:06.625084] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:06.630752] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:06.630811] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668108, retry interval=1000000, ret=-4179) [2022-08-08 10:53:07.560351] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:07.629340] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:07.634157] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:07.634180] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:07.653389] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:07.653504] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668107, retry interval=1000000, ret=-4179) [2022-08-08 10:53:07.657440] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:07.657477] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668112, retry interval=1000000, ret=-4179) [2022-08-08 10:53:07.834602] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927187834598, cur_ts_=1659927187812165, delta=22433) [2022-08-08 10:53:08.207936] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10663, time_dist: generate_cluster_version=10661) [2022-08-08 10:53:08.636551] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:08.639735] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34804) [2022-08-08 10:53:08.643505] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=39401, time_dist: do_cluster_heartbeat=10, generate_cluster_version=39388) [2022-08-08 10:53:08.647664] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:08.647692] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:08.650042] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10260) [2022-08-08 10:53:08.652563] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13605, time_dist: generate_cluster_version=13602) [2022-08-08 10:53:08.692112] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:08.692165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668106, retry interval=1000000, ret=-4179) [2022-08-08 10:53:08.701205] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:08.701252] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668111, retry interval=1000000, ret=-4179) [2022-08-08 10:53:09.007033] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12078, time_dist: generate_cluster_version=12073) [2022-08-08 10:53:09.417173] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=685] [dc=0] clock out of order(cur_ts=1659927189417166, cur_ts_=1659927189392952, delta=24214) [2022-08-08 10:53:09.418461] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23071) [2022-08-08 10:53:09.419273] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25159, time_dist: print_stat=6, generate_cluster_version=25146, cluster_service_self_check=4) [2022-08-08 10:53:09.576785] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=38] [dc=0] [trans sampling] (TRACE=begin_ts=1659927189575321 2022-08-08 02:53:09.575321|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:7369460068315757236, inc:84077741, addr:{ip:"192.168.11.128", port:2882}, t:1659927189574000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5B97-0-0, start_time:1659927189575190, snapshot:1659927189495427|[end_task] u=224 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927189575566|[calc_checksum_by_commit] u=24 ret:0, checksum:2296589347, checksum_log_ts:9223372036854775807|[submit_log] u=83 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1017 trans_version:1659927189575592, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299765, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=1414) [2022-08-08 10:53:09.651465] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:09.654939] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:09.654964] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:09.686076] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927189686069, cur_ts_=1659927189653020, delta=33049) [2022-08-08 10:53:09.690971] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:53:09.691021] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:53:09.717943] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:09.719070] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854668105, retry interval=1000000, ret=-4179) [2022-08-08 10:53:09.725706] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=430] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:09.726869] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=1076] [dc=0] task process failed, start retry(max retry time=9223372036854668110, retry interval=1000000, ret=-4179) [2022-08-08 10:53:09.887091] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14671872}) [2022-08-08 10:53:09.901403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:09.929622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.075495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.077545] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927188256671, end_time=1659927190077536, elapsed_time=1820865, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:53:10.241745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:10.273023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:10.295187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.328418] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.332864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.343798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.346573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:10.371855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.382208] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927190382203, cur_ts_=1659927190353678, delta=28525) [2022-08-08 10:53:10.412163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.414545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.443278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.445136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:10.556577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.596503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.656325] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:10.659561] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:10.659619] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:10.738526] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:10.738576] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668104, retry interval=1000000, ret=-4179) [2022-08-08 10:53:10.747000] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:10.747046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:10.747165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668109, retry interval=1000000, ret=-4179) [2022-08-08 10:53:10.781335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:10.820250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=526] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:11.664637] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:11.668484] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:11.668526] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:11.726232] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927191726225, cur_ts_=1659927191700338, delta=25887) [2022-08-08 10:53:11.804043] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:11.804080] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668108, retry interval=1000000, ret=-4179) [2022-08-08 10:53:11.807035] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:11.807074] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668103, retry interval=1000000, ret=-4179) [2022-08-08 10:53:11.844535] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10959) [2022-08-08 10:53:11.844815] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12278, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12266) [2022-08-08 10:53:12.234347] TRACE [TRACE]ob_trans_ctx.cpp:485 [11024][0][YB42C0A80B80-0005E3CF9BC835EA-0-0] [lt=33] [dc=0] [slow trans] (TRACE=begin_ts=1659927156276400 2022-08-08 02:52:36.276400|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:12092677129610097530, inc:84076495, addr:{ip:"192.168.11.128", port:2882}, t:1659927156275451}, pkey:{tid:1100611139453778, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=5 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835E8-0-0, start_time:1659927156276094, snapshot:1659927156246451|[end_task] u=23479 ret:0, is_rollback:false, uref:1073741825|[handle_timeout] u=10080527 ret:0, used:{click:[1, 0, 5]}, uref:1073741826|[handle_timeout] u=208221 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=406433 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=809567 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=1600676 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=3006066 ret:0, used:{click:[1, 0, 5]}, uref:1073741826|[handle_timeout] u=3008458 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=3000263 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=3006193 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=3008359 ret:0, used:{click:[2, 0, 6]}, uref:1073741826|[handle_timeout] u=3002251 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[start_task] u=2136985 ret:0, sql_no:8589934594, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835E9-0-0, start_time:1659927189573113, snapshot:1659927189495427|[end_task] u=140 ret:0, is_rollback:false, uref:1073741825|[handle_timeout] u=867456 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[set_stc] u=1787180 stc:1659927192228661|[exiting] u=5579 uref:1073741824, arg1:3221573953, arg2:0|[sp_commit] u=7 ret:0, is_rollback:false, is_readonly:false, uref:1|[end_trans_cb] u=24 ret:0, param:0, param:true|[destroy] u=8 arg1:140085233547952|total_timeu=35957882) [2022-08-08 10:53:12.576303] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927192576297, cur_ts_=1659927192542548, delta=33749) [2022-08-08 10:53:12.589356] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C52-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:12.629036] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:12.671527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:12.674837] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:12.674882] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:12.810561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810585] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810594] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810600] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810607] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810770] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.810776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4908-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:12.832841] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:12.832881] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668102, retry interval=1000000, ret=-4179) [2022-08-08 10:53:12.840279] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:12.840343] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668107, retry interval=1000000, ret=-4179) [2022-08-08 10:53:13.566778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:13.679437] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:13.683219] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:13.683300] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:13.697753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:13.717586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:13.750905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:13.864121] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:13.864171] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668106, retry interval=1000000, ret=-4179) [2022-08-08 10:53:13.873657] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:13.873723] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668101, retry interval=1000000, ret=-4179) [2022-08-08 10:53:13.914165] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:13.916623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:13.926246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:13.928504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:13.930471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:13.938942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:13.963902] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:14.037691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.063375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:14.075497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.079717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.081842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:14.089618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:14.090086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.117566] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:14.145178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.158611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.172519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.178480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:14.185913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=229] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.189042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:14.196250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:14.196255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:14.199343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.199385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.200246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:14.206788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:14.209792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:14.222696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:14.230037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.258891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:14.274098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.276659] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.309847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:14.315307] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.343614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:14.349295] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.351490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:14.352961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:14.355053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.454708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.459260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:14.459314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:14.519444] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.550988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.551937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.563364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:14.605047] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2C58-0-0] [lt=12] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927194603434 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927194603424 abs_expired_time=1659927196534402 stmt_timeout=1930978 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21588414 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927194535066 start_version=-1 trans_version=1659927194603424 commit_version=0 stmt_start_time=1659927194600182 abs_expired_time=1659927224502884 stmt_timeout=29902702 abs_lock_wait_timeout=1659927224502884 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:10778588560933161312, inc:84077926, addr:"192.168.11.128:2882", t:1659927194600182} checksum_log_ts=9223372036854775807, version=1659927194603434) [2022-08-08 10:53:14.621966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:14.623020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.684861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:14.685955] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:14.689136] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:14.689187] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:14.810632] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:14.888059] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:14.888098] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668105, retry interval=1000000, ret=-4179) [2022-08-08 10:53:14.897311] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:14.897364] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668100, retry interval=1000000, ret=-4179) [2022-08-08 10:53:14.973438] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:15.042485] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927195042480, cur_ts_=1659927195004807, delta=37673) [2022-08-08 10:53:15.048696] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:15.317030] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:53:15.692421] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:15.698520] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:15.698557] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:15.924658] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:15.924758] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668099, retry interval=1000000, ret=-4179) [2022-08-08 10:53:15.929708] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:15.929772] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668104, retry interval=1000000, ret=-4179) [2022-08-08 10:53:16.700640] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:16.703746] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:16.703783] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:16.928964] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] clock out of order(cur_ts=1659927196928957, cur_ts_=1659927196896536, delta=32421) [2022-08-08 10:53:16.953756] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:16.953801] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668103, retry interval=1000000, ret=-4179) [2022-08-08 10:53:16.964365] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:16.964651] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=192] [dc=0] task process failed, start retry(max retry time=9223372036854668098, retry interval=1000000, ret=-4179) [2022-08-08 10:53:17.706046] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:17.709228] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:17.709253] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:17.978136] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:17.978423] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=227] [dc=0] task process failed, start retry(max retry time=9223372036854668102, retry interval=1000000, ret=-4179) [2022-08-08 10:53:17.988175] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:17.988268] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668097, retry interval=1000000, ret=-4179) [2022-08-08 10:53:18.671881] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927198671875, cur_ts_=1659927198640479, delta=31396) [2022-08-08 10:53:18.711915] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:18.716836] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:18.716886] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:18.998299] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:18.998334] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668101, retry interval=1000000, ret=-4179) [2022-08-08 10:53:19.018200] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:19.018236] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668096, retry interval=1000000, ret=-4179) [2022-08-08 10:53:19.652260] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:53:19.652281] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:53:19.720707] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:19.725756] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:19.727459] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=1682] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:19.823016] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927199823010, cur_ts_=1659927199799271, delta=23739) [2022-08-08 10:53:19.903174] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14676192}) [2022-08-08 10:53:19.918206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:19.949218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.049880] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:20.049947] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668100, retry interval=1000000, ret=-4179) [2022-08-08 10:53:20.061260] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:20.061323] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668095, retry interval=1000000, ret=-4179) [2022-08-08 10:53:20.095881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.097199] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927198254772, end_time=1659927200097187, elapsed_time=1842415, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:53:20.260519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:20.290010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:20.311078] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.346904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.348318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.369140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:20.369218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.390113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.428701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=668] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.429730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.459980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.463179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:20.578540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.619189] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.624446] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C6A-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.624482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C6A-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.624540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C6A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.624616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C6A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.628381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.628409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.628425] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:20.729209] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:20.732536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:20.732582] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:20.767885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:20.799161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:20.835668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:21.072050] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:21.072117] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668099, retry interval=1000000, ret=-4179) [2022-08-08 10:53:21.083742] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:21.083884] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668094, retry interval=1000000, ret=-4179) [2022-08-08 10:53:21.191222] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927201191216, cur_ts_=1659927201162830, delta=28386) [2022-08-08 10:53:21.735089] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:21.739094] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:21.739130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:22.092926] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:22.092965] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668098, retry interval=1000000, ret=-4179) [2022-08-08 10:53:22.111235] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:22.111548] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=260] [dc=0] task process failed, start retry(max retry time=9223372036854668093, retry interval=1000000, ret=-4179) [2022-08-08 10:53:22.303138] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17290, time_dist: generate_cluster_version=17286) [2022-08-08 10:53:22.321682] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16525, time_dist: generate_cluster_version=16521) [2022-08-08 10:53:22.537884] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927202537875, cur_ts_=1659927202515620, delta=22255) [2022-08-08 10:53:22.642084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C70-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:22.642105] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C70-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:22.648635] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:22.648693] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:22.741421] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:22.744713] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:22.744758] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:23.115575] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:23.115610] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668097, retry interval=1000000, ret=-4179) [2022-08-08 10:53:23.135496] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:23.135636] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] task process failed, start retry(max retry time=9223372036854668092, retry interval=1000000, ret=-4179) [2022-08-08 10:53:23.585398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:23.717362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:23.739768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:23.747779] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:23.751122] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:23.751172] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:23.774158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:23.929562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=758] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:23.939844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:23.945727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:23.948815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:23.950354] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:23.959360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:23.980212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:24.062339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.088118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:24.098510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.105136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.105358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:24.105559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:24.115564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.133421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:24.138897] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:24.139584] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668096, retry interval=1000000, ret=-4179) [2022-08-08 10:53:24.158829] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:24.158882] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668091, retry interval=1000000, ret=-4179) [2022-08-08 10:53:24.160310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.180983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.190994] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.198863] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=177] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:24.201268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.211077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:24.211511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:24.214694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:24.220091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.220572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:24.221641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.232230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:24.233512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:24.242503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:24.251788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.284107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:24.294555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.299948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.334675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:24.336256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.360923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:24.366257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=259] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.367355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:24.376289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.377507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:24.470306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.477689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:24.479001] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:24.543660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.573237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.574990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.581230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:24.638373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:24.643739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.709226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:24.754320] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:24.758632] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=184] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:24.758689] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:24.835613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:24.991868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:25.108705] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:25.158523] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:25.158640] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668095, retry interval=1000000, ret=-4179) [2022-08-08 10:53:25.169970] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10633) [2022-08-08 10:53:25.182082] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:25.182139] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668090, retry interval=1000000, ret=-4179) [2022-08-08 10:53:25.760709] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:25.763808] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:25.763862] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:26.144619] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10229) [2022-08-08 10:53:26.146945] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20115, time_dist: generate_cluster_version=20110) [2022-08-08 10:53:26.190475] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927206190466, cur_ts_=1659927206169311, delta=21155) [2022-08-08 10:53:26.221492] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:26.221530] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668094, retry interval=1000000, ret=-4179) [2022-08-08 10:53:26.238467] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:26.238519] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668089, retry interval=1000000, ret=-4179) [2022-08-08 10:53:26.766140] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:26.770572] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:26.771324] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:27.258717] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:27.258768] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668093, retry interval=1000000, ret=-4179) [2022-08-08 10:53:27.274681] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:27.274746] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668088, retry interval=1000000, ret=-4179) [2022-08-08 10:53:27.774598] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:27.777821] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:27.777879] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:28.021535] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927208021526, cur_ts_=1659927207985078, delta=36448) [2022-08-08 10:53:28.277975] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:28.278051] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668092, retry interval=1000000, ret=-4179) [2022-08-08 10:53:28.297694] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15833) [2022-08-08 10:53:28.298022] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:28.298071] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668087, retry interval=1000000, ret=-4179) [2022-08-08 10:53:28.706206] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13874) [2022-08-08 10:53:28.784563] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:28.791206] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:28.791247] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:29.295527] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:29.316396] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] task process failed, start retry(max retry time=9223372036854668091, retry interval=1000000, ret=-4179) [2022-08-08 10:53:29.321343] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927209321336, cur_ts_=1659927209293154, delta=28182) [2022-08-08 10:53:29.329723] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:29.329758] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668086, retry interval=1000000, ret=-4179) [2022-08-08 10:53:29.658112] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:53:29.658144] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:53:29.793129] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:29.796153] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:29.796194] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:29.820953] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14680512}) [2022-08-08 10:53:29.934557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:29.966123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.002732] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927208254694, end_time=1659927210002718, elapsed_time=1748024, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:53:30.116016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.277713] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:30.312988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:30.333615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.334317] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13365) [2022-08-08 10:53:30.334785] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14985, time_dist: generate_cluster_version=14983) [2022-08-08 10:53:30.339930] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:30.339966] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668090, retry interval=1000000, ret=-4179) [2022-08-08 10:53:30.355498] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:30.355532] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668085, retry interval=1000000, ret=-4179) [2022-08-08 10:53:30.363116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.365262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.385990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:30.386878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.406986] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.448661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.448697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.479904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:30.480184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.594468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.641364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.680101] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659927210679287 2022-08-08 02:53:30.679287|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:8591776378801902887, inc:84078540, addr:{ip:"192.168.11.128", port:2882}, t:1659927210678000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5C01-0-0, start_time:1659927210679170, snapshot:1659927210648542|[end_task] u=121 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927210679425|[calc_checksum_by_commit] u=45 ret:0, checksum:1980217439, checksum_log_ts:9223372036854775807|[submit_log] u=68 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=488 trans_version:1659927210679471, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8299871, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=776) [2022-08-08 10:53:30.680996] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C88-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.681027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C88-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.681036] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C88-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.681068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C88-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.682400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=175] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.682658] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=225] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.682678] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.682779] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:30.783705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:30.798606] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:30.803519] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:30.803556] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:30.823920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:30.851167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:30.897928] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11014, time_dist: generate_cluster_version=11011) [2022-08-08 10:53:31.360389] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:31.360795] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] task process failed, start retry(max retry time=9223372036854668089, retry interval=1000000, ret=-4179) [2022-08-08 10:53:31.375177] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:31.375240] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668084, retry interval=1000000, ret=-4179) [2022-08-08 10:53:31.536468] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927211536460, cur_ts_=1659927211513780, delta=22680) [2022-08-08 10:53:31.687668] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C8B-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:31.687721] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C8B-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:31.689255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2C8B-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:31.805874] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:31.809079] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:31.809143] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:32.389026] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:32.389063] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668088, retry interval=1000000, ret=-4179) [2022-08-08 10:53:32.400791] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:32.400840] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668083, retry interval=1000000, ret=-4179) [2022-08-08 10:53:32.629759] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:32.698000] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:32.698054] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:32.811388] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:32.814876] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:32.814931] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:33.123410] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927213123402, cur_ts_=1659927213084185, delta=39217) [2022-08-08 10:53:33.150255] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26863) [2022-08-08 10:53:33.150865] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=48323, time_dist: do_cluster_heartbeat=4, generate_cluster_version=48316) [2022-08-08 10:53:33.153403] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34869, time_dist: generate_cluster_version=34865) [2022-08-08 10:53:33.409476] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:33.409528] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668087, retry interval=1000000, ret=-4179) [2022-08-08 10:53:33.421964] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:33.422024] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668082, retry interval=1000000, ret=-4179) [2022-08-08 10:53:33.603835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:33.736308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:33.761692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:33.793124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:33.817622] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:33.822312] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:33.822357] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:33.949137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:33.959666] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:33.969940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:33.969979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:33.970125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:33.980324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.001278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:34.083654] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.104069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:34.114353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=303] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.123081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.124708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:34.130560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:34.135100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.157389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:34.177383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.204972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.208031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.218926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.223490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:34.229335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:34.230399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.234709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.240674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:34.244269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.244480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.251130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.256804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:34.261479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.275419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.301734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:34.316570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.316611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.356745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.358331] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.376059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:34.383522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.383604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:34.393971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.396232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:34.438191] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:34.438239] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668086, retry interval=1000000, ret=-4179) [2022-08-08 10:53:34.450313] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:34.450365] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668081, retry interval=1000000, ret=-4179) [2022-08-08 10:53:34.487260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.501149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:34.501235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:34.563573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.591298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.594489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.598656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:34.656616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:34.667725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.707887] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17972, time_dist: do_cluster_heartbeat=8, generate_cluster_version=17962) [2022-08-08 10:53:34.729905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:34.825154] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:34.829342] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:34.829390] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:34.857955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:34.952276] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12700) [2022-08-08 10:53:34.952491] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13614, time_dist: generate_cluster_version=13612) [2022-08-08 10:53:35.011186] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:35.167666] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:35.346073] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:53:35.459054] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:35.459105] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668085, retry interval=1000000, ret=-4179) [2022-08-08 10:53:35.470055] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:35.470127] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854668080, retry interval=1000000, ret=-4179) [2022-08-08 10:53:35.832070] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:35.838388] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:35.838466] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:36.485378] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:36.485415] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668084, retry interval=1000000, ret=-4179) [2022-08-08 10:53:36.494379] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:36.494474] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668079, retry interval=1000000, ret=-4179) [2022-08-08 10:53:36.842673] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:36.846125] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:36.846151] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:37.518944] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:37.519007] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668083, retry interval=1000000, ret=-4179) [2022-08-08 10:53:37.522512] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:37.522574] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854668078, retry interval=1000000, ret=-4179) [2022-08-08 10:53:37.858990] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:37.868116] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:37.868161] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:37.996297] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927217996287, cur_ts_=1659927217970014, delta=26273) [2022-08-08 10:53:38.009953] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13612, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13598) [2022-08-08 10:53:38.331103] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10017) [2022-08-08 10:53:38.331491] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11947, time_dist: do_cluster_heartbeat=11, generate_cluster_version=11933) [2022-08-08 10:53:38.350120] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18973) [2022-08-08 10:53:38.351188] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25884, time_dist: generate_cluster_version=25880) [2022-08-08 10:53:38.562107] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:38.562145] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668082, retry interval=1000000, ret=-4179) [2022-08-08 10:53:38.566412] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:38.566445] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668077, retry interval=1000000, ret=-4179) [2022-08-08 10:53:38.572732] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21318) [2022-08-08 10:53:38.573004] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22470, time_dist: generate_cluster_version=22468) [2022-08-08 10:53:38.650921] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927218650915, cur_ts_=1659927218620729, delta=30186) [2022-08-08 10:53:38.871742] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:38.899037] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21944] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:38.899063] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:39.531918] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927219531852, cur_ts_=1659927219509230, delta=22622) [2022-08-08 10:53:39.592147] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:39.592264] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668081, retry interval=1000000, ret=-4179) [2022-08-08 10:53:39.601516] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=137] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:39.601610] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854668076, retry interval=1000000, ret=-4179) [2022-08-08 10:53:39.655212] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:53:39.655236] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:53:39.862513] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927219862504, cur_ts_=1659927219830514, delta=31990) [2022-08-08 10:53:39.888231] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14684960}) [2022-08-08 10:53:39.901477] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:39.904477] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:39.904514] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:39.949737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:39.984795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.079844] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927218256002, end_time=1659927220079834, elapsed_time=1823832, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:53:40.140590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.298788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:40.329972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:40.349699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.380297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.390051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.410872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.410906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:40.425331] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.466336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.473479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.497654] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.507623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:40.594441] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927220594436, cur_ts_=1659927220572510, delta=21926) [2022-08-08 10:53:40.604559] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10141) [2022-08-08 10:53:40.604964] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12223, time_dist: generate_cluster_version=12221) [2022-08-08 10:53:40.616484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.616937] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23209, time_dist: do_cluster_heartbeat=7, generate_cluster_version=23199) [2022-08-08 10:53:40.634236] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:40.634313] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668080, retry interval=1000000, ret=-4179) [2022-08-08 10:53:40.640612] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:40.640678] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668075, retry interval=1000000, ret=-4179) [2022-08-08 10:53:40.665569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.793176] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17299) [2022-08-08 10:53:40.794799] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19581, time_dist: generate_cluster_version=19577) [2022-08-08 10:53:40.795161] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10929, time_dist: generate_cluster_version=10928) [2022-08-08 10:53:40.804346] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:40.841371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:40.867667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:40.906892] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:40.910553] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:40.910601] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:41.506711] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927221506701, cur_ts_=1659927221486433, delta=20268) [2022-08-08 10:53:41.609873] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11844, time_dist: generate_cluster_version=11839) [2022-08-08 10:53:41.685267] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927221685261, cur_ts_=1659927221664758, delta=20503) [2022-08-08 10:53:41.701351] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:41.701418] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668079, retry interval=1000000, ret=-4179) [2022-08-08 10:53:41.706232] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:41.706271] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668074, retry interval=1000000, ret=-4179) [2022-08-08 10:53:41.734710] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13567) [2022-08-08 10:53:41.921204] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:41.925279] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:41.925331] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:42.742859] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927222742855, cur_ts_=1659927222709455, delta=33400) [2022-08-08 10:53:42.749404] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:42.749472] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668078, retry interval=1000000, ret=-4179) [2022-08-08 10:53:42.767383] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:42.767443] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668073, retry interval=1000000, ret=-4179) [2022-08-08 10:53:42.768795] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=32] [dc=0] [trans sampling] (TRACE=begin_ts=1659927222767940 2022-08-08 02:53:42.767940|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:16169417584495543037, inc:84079007, addr:{ip:"192.168.11.128", port:2882}, t:1659927222767000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7B23-0-0, start_time:1659927222767738, snapshot:1659927222709466|[end_task] u=97 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927222768054|[calc_checksum_by_commit] u=17 ret:0, checksum:1076092853, checksum_log_ts:9223372036854775807|[submit_log] u=51 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=573 trans_version:1659927222767466, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:987372, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=793) [2022-08-08 10:53:42.930161] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:42.933251] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:42.933305] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:43.192025] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927223192016, cur_ts_=1659927223171194, delta=20822) [2022-08-08 10:53:43.582912] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927223582898, cur_ts_=1659927223555417, delta=27481) [2022-08-08 10:53:43.616564] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25467) [2022-08-08 10:53:43.616804] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27771, time_dist: do_cluster_heartbeat=12, generate_cluster_version=27757) [2022-08-08 10:53:43.619585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:43.754383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:43.772344] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:43.772385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668077, retry interval=1000000, ret=-4179) [2022-08-08 10:53:43.781650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:43.790467] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:43.790554] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668072, retry interval=1000000, ret=-4179) [2022-08-08 10:53:43.812937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:43.935410] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:43.938528] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:43.938611] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:43.967471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:43.999065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:43.999207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:43.999799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:43.999951] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927223999948, cur_ts_=1659927223975609, delta=24339) [2022-08-08 10:53:44.000128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:44.000205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.037892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:44.054835] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42848) [2022-08-08 10:53:44.078266] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=186] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=67388, time_dist: generate_cluster_version=67383) [2022-08-08 10:53:44.103822] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927224103813, cur_ts_=1659927224054456, delta=49357) [2022-08-08 10:53:44.104949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.105916] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=50856) [2022-08-08 10:53:44.106583] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=53395, time_dist: generate_cluster_version=53391) [2022-08-08 10:53:44.134565] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.135640] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:44.151516] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.151563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:44.152852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.152883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:44.153095] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16926) [2022-08-08 10:53:44.154059] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19223, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19211) [2022-08-08 10:53:44.186183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:44.197332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.227824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.230132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.237509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.248502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:44.248534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:44.248726] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.259909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.261380] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:44.269420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.270100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.270547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.276699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:44.279898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.292534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.321638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:44.332028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.338867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.371942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.373701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.397253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:44.405223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:44.406232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.418188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:44.419725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.509345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.517791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:44.523890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:44.585752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.606549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.613953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:44.619412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.675521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:44.687104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.745543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:44.798160] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:44.798197] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668076, retry interval=1000000, ret=-4179) [2022-08-08 10:53:44.813885] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:44.813946] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668071, retry interval=1000000, ret=-4179) [2022-08-08 10:53:44.880880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:44.940850] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:44.946417] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:44.946455] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:45.027633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:45.066938] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927225066921, cur_ts_=1659927225036036, delta=30885) [2022-08-08 10:53:45.196443] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16896) [2022-08-08 10:53:45.196796] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18011, time_dist: generate_cluster_version=18008) [2022-08-08 10:53:45.229256] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:45.769821] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] [trans sampling] (TRACE=begin_ts=1659927225768777 2022-08-08 02:53:45.768777|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:16073573521267367242, inc:84079115, addr:{ip:"192.168.11.128", port:2882}, t:1659927225768000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5C4C-0-0, start_time:1659927225768650, snapshot:1659927225717556|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927225768918|[calc_checksum_by_commit] u=21 ret:0, checksum:3070072906, checksum_log_ts:9223372036854775807|[submit_log] u=28 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=734 trans_version:1659927225768940, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8299946, logging:0, pending:0, uref:1|[end_trans_cb] u=54 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=1002) [2022-08-08 10:53:45.817852] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:45.817902] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668075, retry interval=1000000, ret=-4179) [2022-08-08 10:53:45.834132] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:45.834194] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668070, retry interval=1000000, ret=-4179) [2022-08-08 10:53:45.948804] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:45.952457] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:45.952503] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:46.843987] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=252] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:46.844038] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668074, retry interval=1000000, ret=-4179) [2022-08-08 10:53:46.863486] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:46.863537] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668069, retry interval=1000000, ret=-4179) [2022-08-08 10:53:46.955101] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:46.958486] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:46.958547] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:47.761014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CBB-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:47.761039] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CBB-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:47.761045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CBB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:47.761049] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CBB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:47.761055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CBB-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:47.851839] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.851891] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.882194] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.882229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.893986] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:47.894263] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668073, retry interval=1000000, ret=-4179) [2022-08-08 10:53:47.902711] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.902853] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.904924] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:47.904975] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668068, retry interval=1000000, ret=-4179) [2022-08-08 10:53:47.961055] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:47.967079] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:47.967103] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:47.975352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC85730-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.975686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC85730-0-0] [lt=140] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.975709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC85730-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.975717] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC85730-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.975823] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.975932] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.976052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:47.976297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.218792] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10184, time_dist: generate_cluster_version=10183) [2022-08-08 10:53:48.291187] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] clock out of order(cur_ts=1659927228291181, cur_ts_=1659927228255886, delta=35295) [2022-08-08 10:53:48.431782] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10521, time_dist: do_cluster_heartbeat=12, generate_cluster_version=10507) [2022-08-08 10:53:48.473400] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659927228473375, cur_ts_=1659927228436891, delta=36484) [2022-08-08 10:53:48.480029] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10639, time_dist: generate_cluster_version=10636) [2022-08-08 10:53:48.596456] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.597482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.597525] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.597538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.627877] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11034, time_dist: generate_cluster_version=11032) [2022-08-08 10:53:48.833953] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10495][0][YB42C0A80B80-0005E3CFCBC85AD0-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1752 is_safe_read=false read_snapshot=1659927228833029 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927228833019 abs_expired_time=1659927230763428 stmt_timeout=1930409 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21588895 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927228770976 start_version=-1 trans_version=1659927228833019 commit_version=0 stmt_start_time=1659927228822964 abs_expired_time=1659927258732326 stmt_timeout=29909362 abs_lock_wait_timeout=1659927258732326 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:6317799443407204364, inc:84080721, addr:"192.168.11.128:2882", t:1659927228822964} checksum_log_ts=9223372036854775807, version=1659927228833029) [2022-08-08 10:53:48.888144] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17075) [2022-08-08 10:53:48.894743] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28812, time_dist: generate_cluster_version=28809) [2022-08-08 10:53:48.903387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.903499] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:48.966072] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:48.966105] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668067, retry interval=1000000, ret=-4179) [2022-08-08 10:53:48.970141] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:48.970169] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668072, retry interval=1000000, ret=-4179) [2022-08-08 10:53:48.973425] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:48.979590] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:48.979615] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:48.986367] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19505) [2022-08-08 10:53:49.006453] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19345) [2022-08-08 10:53:49.013888] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17889, time_dist: generate_cluster_version=17887) [2022-08-08 10:53:49.581879] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927229581873, cur_ts_=1659927229549764, delta=32109) [2022-08-08 10:53:49.658646] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:53:49.658664] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:53:49.666360] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11135, time_dist: do_cluster_heartbeat=3, generate_cluster_version=11129) [2022-08-08 10:53:49.690211] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927229690202, cur_ts_=1659927229666101, delta=24101) [2022-08-08 10:53:49.690482] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25153, time_dist: generate_cluster_version=25152) [2022-08-08 10:53:49.806736] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927229806730, cur_ts_=1659927229774154, delta=32576) [2022-08-08 10:53:49.889810] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18157) [2022-08-08 10:53:49.890031] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19902, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19891) [2022-08-08 10:53:49.900191] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10342) [2022-08-08 10:53:49.900409] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29567, time_dist: do_cluster_heartbeat=5, generate_cluster_version=29560) [2022-08-08 10:53:49.961235] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927229961229, cur_ts_=1659927229927777, delta=33452) [2022-08-08 10:53:49.978180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:49.986290] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:49.997079] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:49.997131] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:50.003298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.077770] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:50.077806] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668071, retry interval=1000000, ret=-4179) [2022-08-08 10:53:50.080750] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:50.080788] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668066, retry interval=1000000, ret=-4179) [2022-08-08 10:53:50.084574] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17799, time_dist: generate_cluster_version=17795) [2022-08-08 10:53:50.169308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.235750] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14689344}) [2022-08-08 10:53:50.295175] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25671, time_dist: generate_cluster_version=25667) [2022-08-08 10:53:50.316671] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927230316665, cur_ts_=1659927230296534, delta=20131) [2022-08-08 10:53:50.325096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:50.360580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:50.371798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.397180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.411599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.431485] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16695) [2022-08-08 10:53:50.432009] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19938, time_dist: do_cluster_heartbeat=7, generate_cluster_version=19928) [2022-08-08 10:53:50.435437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.435830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:50.450677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.461416] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927228291186, end_time=1659927230461402, elapsed_time=2170216, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:53:50.490828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.495557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=226] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.507409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86270-0-0] [lt=157] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:50.507456] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86270-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:50.507862] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86270-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:50.507998] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:50.513370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.523642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=308] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:50.632320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.700588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.821082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:50.859281] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927230859262, cur_ts_=1659927230830029, delta=29233) [2022-08-08 10:53:50.860827] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2CC6-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927230830100 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927230830029 abs_expired_time=1659927232790288 stmt_timeout=1960259 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21588928 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927230830069 start_version=-1 trans_version=1659927230830070 commit_version=0 stmt_start_time=1659927230830029 abs_expired_time=1659927260754235 stmt_timeout=29924206 abs_lock_wait_timeout=1659927260754235 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:14867251205653999865, inc:84083172, addr:"192.168.11.128:2882", t:1659927230830029} checksum_log_ts=9223372036854775807, version=1659927230830100) [2022-08-08 10:53:50.861300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:50.872511] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42678) [2022-08-08 10:53:50.872748] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=618] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=60413, time_dist: generate_cluster_version=60410) [2022-08-08 10:53:50.874315] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=44471, time_dist: do_cluster_heartbeat=10, generate_cluster_version=44459) [2022-08-08 10:53:50.886942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:50.972063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC8641A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:50.998689] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:51.004945] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:51.004995] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:51.113300] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:51.113365] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668065, retry interval=1000000, ret=-4179) [2022-08-08 10:53:51.114213] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:51.114245] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668070, retry interval=1000000, ret=-4179) [2022-08-08 10:53:51.622017] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927231621985, cur_ts_=1659927231600086, delta=21899) [2022-08-08 10:53:51.696141] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:51.851039] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] [trans sampling] (TRACE=begin_ts=1659927231850046 2022-08-08 02:53:51.850046|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:6388586617630548544, inc:84083268, addr:{ip:"192.168.11.128", port:2882}, t:1659927231848000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABA38-0-0, start_time:1659927231849940, snapshot:1659927231821579|[end_task] u=155 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927231850219|[calc_checksum_by_commit] u=22 ret:0, checksum:661133178, checksum_log_ts:9223372036854775807|[submit_log] u=27 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=685 trans_version:1659927231850239, uref:1073741825|[exiting] u=20 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8314569, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=954) [2022-08-08 10:53:52.008994] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:52.012216] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:52.012262] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:52.116256] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FC6-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:52.116295] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FC6-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:52.150715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:52.150866] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:52.155640] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:52.155707] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854668064, retry interval=1000000, ret=-4179) [2022-08-08 10:53:52.157479] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:52.157572] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] task process failed, start retry(max retry time=9223372036854668069, retry interval=1000000, ret=-4179) [2022-08-08 10:53:52.565545] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24959, time_dist: generate_cluster_version=24956) [2022-08-08 10:53:52.571981] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927232571972, cur_ts_=1659927232539899, delta=32073) [2022-08-08 10:53:52.630019] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=110] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:53.014824] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:53.018039] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:53.018074] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:53.165209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:53.165586] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=363] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:53.182121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:53.182263] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:53.184324] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:53.184361] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668063, retry interval=1000000, ret=-4179) [2022-08-08 10:53:53.190251] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:53.190421] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854668068, retry interval=1000000, ret=-4179) [2022-08-08 10:53:53.267817] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:53:53.644540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:53.765424] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.770042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:53.772414] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.777827] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.783045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.791158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.808521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:53.808619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.819202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:53.831986] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:53.984017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.015292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.015342] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:54.020923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.022166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:54.023255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.023367] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:54.026723] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:54.026760] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:54.053424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:54.126597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.156696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.159413] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:54.169095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.169641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.169703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.176859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:54.189648] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:54.213006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.213037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:54.225875] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:54.225935] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668062, retry interval=1000000, ret=-4179) [2022-08-08 10:53:54.246399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.246768] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=289] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:54.247117] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668067, retry interval=1000000, ret=-4179) [2022-08-08 10:53:54.255158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.257367] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.268124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.270552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:53:54.272422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.283064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.285866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.291182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.293072] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:53:54.294667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.294719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=286] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.296128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.310807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.340691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.348533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.360477] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.389282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.390540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.413628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.422679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.423729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:53:54.435967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:54.450961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.534716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.540089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:54.544969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:53:54.605279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.625973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.630468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:54.638260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.700721] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:53:54.708881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:54.767843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:53:54.864778] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBB67-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:54.903089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:55.030101] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:55.039080] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:55.039124] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:55.044708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:53:55.247373] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11258) [2022-08-08 10:53:55.274162] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:55.274195] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668061, retry interval=1000000, ret=-4179) [2022-08-08 10:53:55.285139] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=49713, time_dist: do_cluster_heartbeat=8, generate_cluster_version=49700) [2022-08-08 10:53:55.306009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:55.306986] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:55.312975] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=1752] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:55.313029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668066, retry interval=1000000, ret=-4179) [2022-08-08 10:53:55.324114] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10197) [2022-08-08 10:53:55.330550] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:55.378115] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:53:56.045634] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:56.049450] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:56.049499] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:56.307758] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15816) [2022-08-08 10:53:56.315556] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1822] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24654, time_dist: generate_cluster_version=24648) [2022-08-08 10:53:56.363161] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:56.363286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:56.377676] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:56.377714] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668065, retry interval=1000000, ret=-4179) [2022-08-08 10:53:56.399590] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:56.399654] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:56.403940] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:56.403981] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668060, retry interval=1000000, ret=-4179) [2022-08-08 10:53:56.902380] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CD8-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:56.902403] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CD8-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:56.902985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CD8-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:56.904497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CD8-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:56.904666] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:56.904694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:56.928669] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16728) [2022-08-08 10:53:56.938480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBB9D-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:57.054001] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:57.057407] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:57.057531] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:57.385813] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:57.385888] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:57.414950] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:57.414984] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668064, retry interval=1000000, ret=-4179) [2022-08-08 10:53:57.434691] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:57.434741] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668059, retry interval=1000000, ret=-4179) [2022-08-08 10:53:57.753142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E6EE0-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:57.753213] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E6EE0-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:57.753541] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:53:57.895123] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15088, time_dist: generate_cluster_version=15085) [2022-08-08 10:53:58.059945] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:58.063226] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:58.063264] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:58.448878] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:58.485637] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:58.485671] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668063, retry interval=1000000, ret=-4179) [2022-08-08 10:53:58.490162] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20713) [2022-08-08 10:53:58.491414] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22764, time_dist: print_stat=7, do_cluster_heartbeat=2, generate_cluster_version=22746, cluster_service_self_check=5) [2022-08-08 10:53:58.500607] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=42225, time_dist: do_cluster_heartbeat=13, generate_cluster_version=42209) [2022-08-08 10:53:58.509067] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:58.509102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:58.511633] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:58.511689] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668058, retry interval=1000000, ret=-4179) [2022-08-08 10:53:58.718671] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927238718660, cur_ts_=1659927238693461, delta=25199) [2022-08-08 10:53:58.751828] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22447) [2022-08-08 10:53:58.753975] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31557, time_dist: generate_cluster_version=31554) [2022-08-08 10:53:58.889144] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927238889134, cur_ts_=1659927238854024, delta=35110) [2022-08-08 10:53:58.937817] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37346) [2022-08-08 10:53:58.938304] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=38920, time_dist: do_cluster_heartbeat=12, generate_cluster_version=38903) [2022-08-08 10:53:58.948774] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10863) [2022-08-08 10:53:58.949032] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14663, time_dist: do_cluster_heartbeat=7, generate_cluster_version=14654) [2022-08-08 10:53:59.065679] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:53:59.072611] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:53:59.072780] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:53:59.125546] WARN [SERVER] load_info_item (ob_backup_operator.cpp:1396) [10495][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] backup info is not exist yet, wait later(ret=-9032, tenant_id=1, sql=SELECT name, value FROM __all_tenant_backup_info WHERE name = 'tenant_name_backup_schema_version' AND tenant_id = 1) [2022-08-08 10:53:59.125601] WARN [SERVER] get_tenant_name_backup_schema_version (ob_backup_operator.cpp:1487) [10495][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] tenant backup info not exist(ret=-9032, backup_schema_version=0) [2022-08-08 10:53:59.125768] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBC64-0-0] [lt=238] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.125783] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBC64-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.125788] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBC64-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.125793] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBC64-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.127596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.127626] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.127632] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.127636] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.127652] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.127660] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.127664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86420-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.182716] WARN [SERVER] load_info_item (ob_backup_operator.cpp:1396) [10495][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] backup info is not exist yet, wait later(ret=-9032, tenant_id=1001, sql=SELECT name, value FROM __all_tenant_backup_info WHERE name = 'backup_schema_version' AND tenant_id = 1001) [2022-08-08 10:53:59.182813] WARN [SERVER] get_backup_schema_version (ob_backup_operator.cpp:1462) [10495][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] tenant backup info not exist(ret=-9032, backup_schema_version=0) [2022-08-08 10:53:59.276708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.304040] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.312582] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:53:59.516807] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:59.516914] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668062, retry interval=1000000, ret=-4179) [2022-08-08 10:53:59.544071] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] clock out of order(cur_ts=1659927239544066, cur_ts_=1659927239511582, delta=32484) [2022-08-08 10:53:59.564080] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.564337] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=242] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:53:59.591138] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:53:59.591269] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668057, retry interval=1000000, ret=-4179) [2022-08-08 10:53:59.611974] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15894) [2022-08-08 10:53:59.660883] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:53:59.660920] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:53:59.930146] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927239930136, cur_ts_=1659927239901716, delta=28420) [2022-08-08 10:53:59.931420] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=27345) [2022-08-08 10:53:59.940812] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10504) [2022-08-08 10:53:59.941088] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37787, time_dist: do_cluster_heartbeat=9, generate_cluster_version=37775) [2022-08-08 10:54:00.002516] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.018810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.075461] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:00.080341] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:00.080387] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:00.184816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=6369] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.231986] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14693728}) [2022-08-08 10:54:00.344534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=190] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:00.381852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:00.393436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.418604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.428409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.436325] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927238257688, end_time=1659927240436312, elapsed_time=2178624, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:54:00.460832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:00.461013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.496256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.496515] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927240496509, cur_ts_=1659927240466016, delta=30493) [2022-08-08 10:54:00.509403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.513486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.515400] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13254) [2022-08-08 10:54:00.515885] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19090, time_dist: print_stat=6, do_cluster_heartbeat=2, generate_cluster_version=19078) [2022-08-08 10:54:00.516431] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19689, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19672) [2022-08-08 10:54:00.526024] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C494E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.526098] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C494E-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.526292] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C494E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.526498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C494E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.526664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C494E-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.529755] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.529872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.545415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.545872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:00.580990] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.581040] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.582947] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:00.583026] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668061, retry interval=1000000, ret=-4179) [2022-08-08 10:54:00.595200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FD2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.595810] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FD2-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:00.651353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.654201] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927240654197, cur_ts_=1659927240633336, delta=20861) [2022-08-08 10:54:00.678990] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:00.679050] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668056, retry interval=1000000, ret=-4179) [2022-08-08 10:54:00.719476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.838805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:00.885648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:00.908458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:01.014088] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2085] [dc=0] clock out of order(cur_ts=1659927241014079, cur_ts_=1659927240979840, delta=34239) [2022-08-08 10:54:01.035106] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CE4-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:01.035131] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CE4-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:01.035752] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:01.035782] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:01.086542] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:01.092706] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:01.092759] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:01.383200] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927241383192, cur_ts_=1659927241360820, delta=22372) [2022-08-08 10:54:01.607153] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:01.607557] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:01.611023] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:01.611081] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668060, retry interval=1000000, ret=-4179) [2022-08-08 10:54:01.702697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:01.702732] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:01.704288] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:01.704328] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668055, retry interval=1000000, ret=-4179) [2022-08-08 10:54:02.096855] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:02.102482] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:02.102533] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:02.615072] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FD5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:02.615162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FD5-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:02.615304] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FD5-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:02.637751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:02.638045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:02.639596] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:02.639653] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668059, retry interval=1000000, ret=-4179) [2022-08-08 10:54:02.706502] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4951-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:02.726810] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:02.726850] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668054, retry interval=1000000, ret=-4179) [2022-08-08 10:54:03.104724] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:03.107956] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:03.107993] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:03.662817] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:03.666933] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:03.666983] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668058, retry interval=1000000, ret=-4179) [2022-08-08 10:54:03.729178] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47D2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:03.729219] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47D2-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:03.745225] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:03.745387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:03.747272] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:03.747309] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668053, retry interval=1000000, ret=-4179) [2022-08-08 10:54:03.748091] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10179, time_dist: generate_cluster_version=10176) [2022-08-08 10:54:03.792210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:03.834175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:03.854778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.000687] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.032177] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.038667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:04.039580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.041877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.042984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:04.076663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:04.077484] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927244077480, cur_ts_=1659927244053215, delta=24265) [2022-08-08 10:54:04.110344] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:04.113562] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:04.113611] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:04.149970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.181183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.181393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:04.191618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.191778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.191792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.209486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:04.232425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:04.235556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.252546] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927244252541, cur_ts_=1659927244227172, delta=25369) [2022-08-08 10:54:04.276042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.276675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=178] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.276919] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.285923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.286124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:04.298435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.300857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.306712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.311017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.317077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:04.317204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.318926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.319187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.352602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.360188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.370680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.381275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.413955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.414116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.438587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.439498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.445257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:04.457313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:04.469819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.553785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.564009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:04.564182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:04.623515] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.646058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.652498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:04.662210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.671612] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:04.671642] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:04.688480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:04.688687] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:04.689891] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:04.690039] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668057, retry interval=1000000, ret=-4179) [2022-08-08 10:54:04.723241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:04.728650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:04.767746] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:04.767810] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:04.769054] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:04.769104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668052, retry interval=1000000, ret=-4179) [2022-08-08 10:54:04.791134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:04.925229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:05.063574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:05.116020] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:05.120994] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:05.121037] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:05.397669] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:05.702459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:05.702545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:05.722740] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:05.725820] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:05.725874] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668056, retry interval=1000000, ret=-4179) [2022-08-08 10:54:05.791429] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:05.791472] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668051, retry interval=1000000, ret=-4179) [2022-08-08 10:54:06.123422] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:06.126973] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:06.127001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:06.758550] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:06.758589] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668055, retry interval=1000000, ret=-4179) [2022-08-08 10:54:06.809665] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:06.810336] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=604] [dc=0] task process failed, start retry(max retry time=9223372036854668050, retry interval=1000000, ret=-4179) [2022-08-08 10:54:07.101386] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CF6-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:07.101530] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CF6-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:07.101773] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:07.101881] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:07.102015] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:07.102071] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:07.130520] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:07.135180] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:07.135237] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:07.228091] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927247228083, cur_ts_=1659927247206178, delta=21905) [2022-08-08 10:54:07.761978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4817-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:07.779853] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:07.781490] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:07.781529] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668054, retry interval=1000000, ret=-4179) [2022-08-08 10:54:07.829116] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:07.830881] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:07.830946] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668049, retry interval=1000000, ret=-4179) [2022-08-08 10:54:08.109516] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CF9-0-0] [lt=528] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.109554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CF9-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.109755] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CF9-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.109848] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CF9-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.110014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.110042] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.110127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:08.138111] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:08.141951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:08.142019] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:08.303043] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11991, time_dist: do_cluster_heartbeat=9, generate_cluster_version=11979) [2022-08-08 10:54:08.389116] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13687, time_dist: generate_cluster_version=13686) [2022-08-08 10:54:08.807063] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:08.807097] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668053, retry interval=1000000, ret=-4179) [2022-08-08 10:54:08.834751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FDE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:08.854835] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:08.856458] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=194] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:08.856504] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668048, retry interval=1000000, ret=-4179) [2022-08-08 10:54:09.146740] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:09.150133] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:09.150160] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:09.325879] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] clock out of order(cur_ts=1659927249325870, cur_ts_=1659927249300328, delta=25542) [2022-08-08 10:54:09.446424] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927249446415, cur_ts_=1659927249418856, delta=27559) [2022-08-08 10:54:09.667202] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:54:09.667253] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:54:09.807995] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927249807987, cur_ts_=1659927249781250, delta=26737) [2022-08-08 10:54:09.835255] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:09.835293] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668052, retry interval=1000000, ret=-4179) [2022-08-08 10:54:09.876382] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:09.880682] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:09.885441] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:09.885682] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668047, retry interval=1000000, ret=-4179) [2022-08-08 10:54:09.935368] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927249935358, cur_ts_=1659927249910872, delta=24486) [2022-08-08 10:54:09.937280] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=213] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14698112}) [2022-08-08 10:54:09.942549] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=326] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19505) [2022-08-08 10:54:09.945308] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23164, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23149) [2022-08-08 10:54:10.021270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=234] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.043905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=6429] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.136806] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927248259187, end_time=1659927250136793, elapsed_time=1877606, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:54:10.143816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CFF-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:10.143930] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CFF-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:10.144175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2CFF-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:10.144371] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:10.144497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:10.152246] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:10.155753] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:10.156066] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=297] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:10.209722] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.364839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:10.397731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:10.418435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.436217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.447617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.480296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.481718] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:10.513175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.533882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.535013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.567662] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20114) [2022-08-08 10:54:10.567752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:10.567897] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21053, time_dist: generate_cluster_version=21051) [2022-08-08 10:54:10.581367] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=401] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.669955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=2953] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.742498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.742908] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17932) [2022-08-08 10:54:10.743151] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20100, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20066) [2022-08-08 10:54:10.771699] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28741) [2022-08-08 10:54:10.771940] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=39591, time_dist: do_cluster_heartbeat=10, generate_cluster_version=39579) [2022-08-08 10:54:10.862889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:10.876262] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:10.876576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:10.878041] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:10.878170] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668051, retry interval=1000000, ret=-4179) [2022-08-08 10:54:10.903266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:10.903980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:10.904895] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:10.904944] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668046, retry interval=1000000, ret=-4179) [2022-08-08 10:54:10.928401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:11.159542] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:11.165896] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:11.165934] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:11.563203] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25372) [2022-08-08 10:54:11.563372] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26891, time_dist: do_cluster_heartbeat=10, generate_cluster_version=26879) [2022-08-08 10:54:11.564927] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13130, time_dist: generate_cluster_version=13128) [2022-08-08 10:54:11.890706] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:11.908018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:11.908506] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=458] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:11.909752] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:11.909801] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668050, retry interval=1000000, ret=-4179) [2022-08-08 10:54:11.927820] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:11.927975] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:11.929547] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:11.929597] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668045, retry interval=1000000, ret=-4179) [2022-08-08 10:54:12.157350] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D05-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.157383] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D05-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.157509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D05-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.157681] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D05-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.157835] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.157962] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.158103] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.158264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.164326] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.164440] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.164455] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:12.168210] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:12.174623] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:12.174663] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:12.431905] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13148) [2022-08-08 10:54:12.432154] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14195, time_dist: do_cluster_heartbeat=8, generate_cluster_version=14183) [2022-08-08 10:54:12.642389] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:12.912167] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FE4-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:12.912324] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FE4-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:12.931613] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4960-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:12.931739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4960-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:12.942790] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:12.942828] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668049, retry interval=1000000, ret=-4179) [2022-08-08 10:54:12.950061] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:12.951903] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:12.951965] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854668044, retry interval=1000000, ret=-4179) [2022-08-08 10:54:12.953242] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] [trans sampling] (TRACE=begin_ts=1659927252952526 2022-08-08 02:54:12.952526|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:8676024829468888922, inc:84085176, addr:{ip:"192.168.11.128", port:2882}, t:1659927252952000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=5 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7B5D-0-0, start_time:1659927252952419, snapshot:1659927252942236|[end_task] u=93 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=7 stc:1659927252952634|[calc_checksum_by_commit] u=17 ret:0, checksum:1430498075, checksum_log_ts:9223372036854775807|[submit_log] u=52 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=472 trans_version:1659927252952472, uref:1073741825|[exiting] u=8 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:987430, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233102640|total_timeu=681) [2022-08-08 10:54:13.180122] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:13.183400] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:13.183438] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:13.697218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:13.697790] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927253697786, cur_ts_=1659927253664383, delta=33403) [2022-08-08 10:54:13.811075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:13.852639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:13.877971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:13.989204] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:13.989322] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854668048, retry interval=1000000, ret=-4179) [2022-08-08 10:54:13.990405] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18625, time_dist: generate_cluster_version=18622) [2022-08-08 10:54:13.993055] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:13.993117] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668043, retry interval=1000000, ret=-4179) [2022-08-08 10:54:14.019937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.053549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.057981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=234] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.061288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:14.061735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:14.064040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.093669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:14.167109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.181290] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D0B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.181355] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D0B-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.181552] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D0B-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.181722] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D0B-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.181872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.181996] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.182134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.182248] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:14.185506] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:14.188836] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:14.188904] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:14.197371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.206577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:14.211794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.212295] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.213267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.233102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:14.253547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:14.258442] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=14486] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.300093] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.300179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.300219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.305581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=151] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:14.310537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.321035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.321145] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.324373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.331856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.332321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.334723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=231] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.336445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.341780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:14.376388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.383932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.389721] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:14.389792] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:14.389859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:14.393845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.404240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.430095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.434987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.455811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.463554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.463617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:14.476139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:14.492556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.568605] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927254568504, cur_ts_=1659927254547548, delta=20956) [2022-08-08 10:54:14.571734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.580342] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:14.580795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:14.648421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.669341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.674383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:14.678899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.741663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:14.751567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:14.840415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:14.842813] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2752] [dc=0] clock out of order(cur_ts=1659927254842807, cur_ts_=1659927254811042, delta=31765) [2022-08-08 10:54:14.949875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:15.018488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:15.018696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:15.020023] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:15.020785] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=633] [dc=0] task process failed, start retry(max retry time=9223372036854668047, retry interval=1000000, ret=-4179) [2022-08-08 10:54:15.030111] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:15.030244] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:15.031737] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:15.031787] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668042, retry interval=1000000, ret=-4179) [2022-08-08 10:54:15.087673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:15.191884] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:15.197536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:15.197580] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:15.407806] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:54:15.436532] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:15.436677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:15.465346] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:16.008048] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=50] [dc=0] [trans sampling] (TRACE=begin_ts=1659927256002564 2022-08-08 02:54:16.002564|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:14230071157995030419, inc:84085391, addr:{ip:"192.168.11.128", port:2882}, t:1659927256002000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5CE2-0-0, start_time:1659927256002439, snapshot:1659927255955711|[end_task] u=129 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927256002710|[calc_checksum_by_commit] u=73 ret:0, checksum:117729209, checksum_log_ts:9223372036854775807|[submit_log] u=52 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=4817 trans_version:1659927256002731, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8300096, logging:0, pending:0, uref:1|[end_trans_cb] u=275 ret:0, param:0, param:true|[destroy] u=6 arg1:140085232375600|total_timeu=5392) [2022-08-08 10:54:16.048129] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:16.048165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668046, retry interval=1000000, ret=-4179) [2022-08-08 10:54:16.059036] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:16.059086] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668041, retry interval=1000000, ret=-4179) [2022-08-08 10:54:16.199729] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:16.203158] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:16.203193] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:16.480831] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927256480825, cur_ts_=1659927256458139, delta=22686) [2022-08-08 10:54:17.076868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:17.077995] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:17.078040] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668045, retry interval=1000000, ret=-4179) [2022-08-08 10:54:17.080911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:17.082226] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:17.082288] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668040, retry interval=1000000, ret=-4179) [2022-08-08 10:54:17.205632] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:17.208725] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:17.208767] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:17.594188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:17.594298] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:18.084119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47E7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:18.084136] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47E7-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:18.084174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47E7-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:18.121691] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:18.121725] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668044, retry interval=1000000, ret=-4179) [2022-08-08 10:54:18.129786] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:18.129935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:18.134240] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:18.134298] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668039, retry interval=1000000, ret=-4179) [2022-08-08 10:54:18.210985] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:18.216309] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:18.216348] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:18.687095] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927258687090, cur_ts_=1659927258657586, delta=29504) [2022-08-08 10:54:18.701629] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:18.701710] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:18.701763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:18.701822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:18.993032] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14953) [2022-08-08 10:54:18.995903] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18744, time_dist: generate_cluster_version=18741) [2022-08-08 10:54:19.139272] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4969-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:19.156889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:19.156914] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:19.157378] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:19.157416] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668043, retry interval=1000000, ret=-4179) [2022-08-08 10:54:19.171161] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:19.171222] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668038, retry interval=1000000, ret=-4179) [2022-08-08 10:54:19.179955] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12154) [2022-08-08 10:54:19.210007] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D1A-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:19.210028] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D1A-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:19.210683] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:19.210701] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:19.210710] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:19.210719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:19.218485] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:19.221643] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:19.221665] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:19.403302] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=148] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11637, time_dist: generate_cluster_version=11633) [2022-08-08 10:54:19.661573] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:54:19.661610] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:54:19.813392] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15971, time_dist: do_cluster_heartbeat=5, generate_cluster_version=15962) [2022-08-08 10:54:19.924775] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14702496}) [2022-08-08 10:54:20.016141] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10943, time_dist: generate_cluster_version=10940) [2022-08-08 10:54:20.037904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.067778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.120917] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927258259679, end_time=1659927260120905, elapsed_time=1861226, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:54:20.167159] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.167275] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.173245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47EA-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.173315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47EA-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.173429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47EA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.173717] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47EA-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.186336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.186472] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.188512] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=376] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:20.188571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668042, retry interval=1000000, ret=-4179) [2022-08-08 10:54:20.191401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.191573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:20.195071] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:20.195246] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] task process failed, start retry(max retry time=9223372036854668037, retry interval=1000000, ret=-4179) [2022-08-08 10:54:20.226150] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:20.226451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.229382] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:20.229453] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:20.385436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:20.419592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:20.443511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.458279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.465005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.500013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:20.504036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.537244] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.550568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.557505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.591169] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:20.603922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.691271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.710766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBDD7-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:20.710869] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBDD7-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:20.710880] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBDD7-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:20.710885] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBDD7-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:20.711998] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:20.712040] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:20.739130] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927260739120, cur_ts_=1659927260714101, delta=25019) [2022-08-08 10:54:20.759241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.884422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:20.928265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:20.946742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:21.190840] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:21.191017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF2-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:21.222997] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:21.223029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668036, retry interval=1000000, ret=-4179) [2022-08-08 10:54:21.230561] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:21.230564] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2D20-0-0] [lt=1] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927261222476 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927261222455 abs_expired_time=1659927263159795 stmt_timeout=1937340 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21589352 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927261222465 start_version=-1 trans_version=1659927261222466 commit_version=0 stmt_start_time=1659927261222455 abs_expired_time=1659927291128151 stmt_timeout=29905696 abs_lock_wait_timeout=1659927291128151 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:10126049220388236981, inc:84085791, addr:"192.168.11.128:2882", t:1659927261222455} checksum_log_ts=9223372036854775807, version=1659927261222476) [2022-08-08 10:54:21.230591] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668041, retry interval=1000000, ret=-4179) [2022-08-08 10:54:21.231793] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:21.235529] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:21.235561] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:22.066737] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] [trans sampling] (TRACE=begin_ts=1659927262065954 2022-08-08 02:54:22.065954|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:7978424176788354717, inc:84085837, addr:{ip:"192.168.11.128", port:2882}, t:1659927262064000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABACE-0-0, start_time:1659927262065806, snapshot:1659927262045692|[end_task] u=140 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927262066114|[calc_checksum_by_commit] u=17 ret:0, checksum:1568465361, checksum_log_ts:9223372036854775807|[submit_log] u=49 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=469 trans_version:1659927262064948, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8314719, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=743) [2022-08-08 10:54:22.237930] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:22.238231] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12869, time_dist: generate_cluster_version=12866) [2022-08-08 10:54:22.252017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:22.252113] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:22.253464] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:22.253497] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668035, retry interval=1000000, ret=-4179) [2022-08-08 10:54:22.257996] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:22.258033] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:22.273548] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:22.273715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:22.275759] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:22.275833] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668040, retry interval=1000000, ret=-4179) [2022-08-08 10:54:22.799392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBE02-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:22.799436] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBE02-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:22.799954] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:22.800257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=270] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:22.800274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:22.800317] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 10:54:23.264683] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:23.275538] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:23.275594] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:23.277945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:23.292309] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:23.292343] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668034, retry interval=1000000, ret=-4179) [2022-08-08 10:54:23.298160] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:23.299397] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:23.299446] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668039, retry interval=1000000, ret=-4179) [2022-08-08 10:54:23.715933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:23.829398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:23.871710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:23.900187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.041211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.076481] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.078926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:24.082318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.082394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:24.087007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.113780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:24.182658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.221749] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.231660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:24.232233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.235128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.235179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.252003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:24.270830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:24.278733] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:24.281352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D29-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:24.282416] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:24.282452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:24.283723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.317026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.317076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.321888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.323898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:24.325041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:24.325260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:24.326643] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:24.326676] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668033, retry interval=1000000, ret=-4179) [2022-08-08 10:54:24.327531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.328392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:24.328451] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:24.330242] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:24.330333] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854668038, retry interval=1000000, ret=-4179) [2022-08-08 10:54:24.337836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.344196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.350771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.351218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.351289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.356108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.359630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.361286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:24.402006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.402794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.414670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.425751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.454465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.457861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.478714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.478749] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.484633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:24.521446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.522098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:24.522135] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=369] [dc=0] clock out of order(cur_ts=1659927264522131, cur_ts_=1659927264496525, delta=25606) [2022-08-08 10:54:24.590794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.601122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:24.601176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:24.673049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=562] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.685436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.696222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:24.703792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.758133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:24.776670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:24.863581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=2034] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:24.971591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:25.111435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:25.215347] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14221) [2022-08-08 10:54:25.215601] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15224, time_dist: generate_cluster_version=15222) [2022-08-08 10:54:25.285572] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:25.291523] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:25.291563] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:25.332620] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF8-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.332648] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF8-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.332654] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF8-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.332664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF8-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.332695] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC5FF8-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.358030] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:25.358067] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668037, retry interval=1000000, ret=-4179) [2022-08-08 10:54:25.360944] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.361144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:25.362456] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:25.362490] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668032, retry interval=1000000, ret=-4179) [2022-08-08 10:54:25.563097] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:26.217885] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927266217874, cur_ts_=1659927266196735, delta=21139) [2022-08-08 10:54:26.248641] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33390) [2022-08-08 10:54:26.248909] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=37220, time_dist: do_cluster_heartbeat=9, generate_cluster_version=37209) [2022-08-08 10:54:26.294427] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:26.330813] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927266330806, cur_ts_=1659927266292874, delta=37932) [2022-08-08 10:54:26.337452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:26.337496] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:26.341472] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:26.370295] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.371042] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.371809] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=628] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.371920] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.371985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.371992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.372160] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47F3-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:26.447016] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927266447005, cur_ts_=1659927266424437, delta=22568) [2022-08-08 10:54:26.449529] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26126) [2022-08-08 10:54:26.449979] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27557, time_dist: do_cluster_heartbeat=14, generate_cluster_version=27540) [2022-08-08 10:54:26.478081] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11288] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:26.478141] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668036, retry interval=1000000, ret=-4179) [2022-08-08 10:54:26.481007] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:26.481038] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668031, retry interval=1000000, ret=-4179) [2022-08-08 10:54:26.494452] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13519) [2022-08-08 10:54:26.495955] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15800, time_dist: generate_cluster_version=15797) [2022-08-08 10:54:26.906887] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23659, time_dist: generate_cluster_version=23655) [2022-08-08 10:54:26.917232] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10531) [2022-08-08 10:54:26.918378] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30134, time_dist: generate_cluster_version=30129) [2022-08-08 10:54:27.221026] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19617) [2022-08-08 10:54:27.221224] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21198, time_dist: do_cluster_heartbeat=28, generate_cluster_version=21168) [2022-08-08 10:54:27.354361] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:27.364194] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:27.364282] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:27.531071] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:27.531124] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668035, retry interval=1000000, ret=-4179) [2022-08-08 10:54:27.531820] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:27.531852] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668030, retry interval=1000000, ret=-4179) [2022-08-08 10:54:27.681526] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10513) [2022-08-08 10:54:27.682295] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12196, time_dist: generate_cluster_version=12193) [2022-08-08 10:54:27.946406] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927267946394, cur_ts_=1659927267911476, delta=34918) [2022-08-08 10:54:28.226709] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=63] [dc=0] [trans sampling] (TRACE=begin_ts=1659927268222941 2022-08-08 02:54:28.222941|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:5397838162833598360, inc:84086311, addr:{ip:"192.168.11.128", port:2882}, t:1659927268221000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5D1F-0-0, start_time:1659927268222722, snapshot:1659927268179929|[end_task] u=138 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927268223099|[calc_checksum_by_commit] u=22 ret:0, checksum:1393668474, checksum_log_ts:9223372036854775807|[submit_log] u=32 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1908 trans_version:1659927268223122, uref:1073741825|[exiting] u=351 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=5 ret:0, log_type:64, log_id:8300157, logging:0, pending:0, uref:1|[end_trans_cb] u=964 ret:0, param:0, param:true|[destroy] u=21 arg1:140085232375600|total_timeu=3464) [2022-08-08 10:54:28.377654] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:28.386608] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:28.386638] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:28.626422] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=312] [dc=0] clock out of order(cur_ts=1659927268626416, cur_ts_=1659927268606107, delta=20309) [2022-08-08 10:54:28.639752] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:28.639805] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668029, retry interval=1000000, ret=-4179) [2022-08-08 10:54:28.641602] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:28.641703] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668034, retry interval=1000000, ret=-4179) [2022-08-08 10:54:28.809525] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] clock out of order(cur_ts=1659927268809515, cur_ts_=1659927268789131, delta=20384) [2022-08-08 10:54:29.390934] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:29.398636] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:29.398664] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:29.402844] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:29.402932] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:29.403594] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:29.403679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:29.403697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:29.423486] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13765) [2022-08-08 10:54:29.656003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:29.656024] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:29.705052] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23106) [2022-08-08 10:54:29.705544] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=408] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29668, time_dist: do_cluster_heartbeat=8, generate_cluster_version=29657) [2022-08-08 10:54:29.727114] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:54:29.727134] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:54:29.731375] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26281) [2022-08-08 10:54:29.741235] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=63575, time_dist: do_cluster_heartbeat=8, generate_cluster_version=63564) [2022-08-08 10:54:29.745251] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:29.747666] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:29.781154] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:29.781192] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668033, retry interval=1000000, ret=-4179) [2022-08-08 10:54:29.795272] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=183] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:29.795696] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] task process failed, start retry(max retry time=9223372036854668028, retry interval=1000000, ret=-4179) [2022-08-08 10:54:30.062626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.086628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.247075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.315129] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927270315124, cur_ts_=1659927270289199, delta=25925) [2022-08-08 10:54:30.320937] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=789] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13000, time_dist: generate_cluster_version=12994) [2022-08-08 10:54:30.325029] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15666, time_dist: generate_cluster_version=15662) [2022-08-08 10:54:30.400911] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:30.405389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:30.409449] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:30.409475] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:30.446099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:30.469274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.477733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.481567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.495521] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13764) [2022-08-08 10:54:30.495871] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28939, time_dist: do_cluster_heartbeat=8, generate_cluster_version=28928) [2022-08-08 10:54:30.497636] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31432, time_dist: do_cluster_heartbeat=56, generate_cluster_version=31370) [2022-08-08 10:54:30.524604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.524805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:30.552784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.573686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.574447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.610686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:30.621133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.709684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.727833] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14706816}) [2022-08-08 10:54:30.792309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.798692] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927270798685, cur_ts_=1659927270777579, delta=21106) [2022-08-08 10:54:30.812681] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4838-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:30.875554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:30.875604] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:30.903532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:30.940779] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=172] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16055, time_dist: do_cluster_heartbeat=7, generate_cluster_version=16046) [2022-08-08 10:54:30.960800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:30.962195] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:30.964818] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24628) [2022-08-08 10:54:30.965879] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:30.965917] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668027, retry interval=1000000, ret=-4179) [2022-08-08 10:54:30.966245] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29079, time_dist: generate_cluster_version=29076) [2022-08-08 10:54:30.971840] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:30.971872] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668032, retry interval=1000000, ret=-4179) [2022-08-08 10:54:30.977031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:31.050763] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927268259844, end_time=1659927271050750, elapsed_time=2790906, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:54:31.244532] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927271241576 2022-08-08 02:54:31.241576|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:10838004688807955427, inc:84086422, addr:{ip:"192.168.11.128", port:2882}, t:1659927271241000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=10 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5D2E-0-0, start_time:1659927271241346, snapshot:1659927271166775|[end_task] u=140 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927271241740|[calc_checksum_by_commit] u=21 ret:0, checksum:4003674205, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2422 trans_version:1659927271241762, uref:1073741825|[exiting] u=80 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=5 ret:0, log_type:64, log_id:8300172, logging:0, pending:0, uref:1|[end_trans_cb] u=27 ret:0, param:0, param:true|[destroy] u=10 arg1:140085232375600|total_timeu=2786) [2022-08-08 10:54:31.434024] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:31.438056] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:31.438130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:32.009740] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:32.009899] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668031, retry interval=1000000, ret=-4179) [2022-08-08 10:54:32.013895] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:32.013933] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668026, retry interval=1000000, ret=-4179) [2022-08-08 10:54:32.440861] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:32.447221] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:32.447286] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:32.643339] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:32.947581] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927272947576, cur_ts_=1659927272920286, delta=27290) [2022-08-08 10:54:33.012458] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.012510] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.012653] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.012723] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.012871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.013030] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.013227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.014573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C483B-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.015592] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.015643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.015651] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016022] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=356] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016048] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016168] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=101] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016180] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016190] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016205] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016225] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.016432] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FC-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:33.044534] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:33.044811] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854668030, retry interval=1000000, ret=-4179) [2022-08-08 10:54:33.046608] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:33.046684] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854668025, retry interval=1000000, ret=-4179) [2022-08-08 10:54:33.449960] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:33.453592] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:33.453647] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:33.737718] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:33.850489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:33.893522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:33.917083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.065542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.080228] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:34.080265] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668029, retry interval=1000000, ret=-4179) [2022-08-08 10:54:34.085083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:34.086237] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:34.086306] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668024, retry interval=1000000, ret=-4179) [2022-08-08 10:54:34.094739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:34.100338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.105138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:34.105695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.114150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.131508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:34.197996] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.246630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.248053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.249106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:34.259208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.259299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.260298] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] [trans sampling] (TRACE=begin_ts=1659927274259261 2022-08-08 02:54:34.259261|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:12036741817385241529, inc:84086542, addr:{ip:"192.168.11.128", port:2882}, t:1659927274259000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABB0B-0-0, start_time:1659927274259143, snapshot:1659927274183182|[end_task] u=162 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927274259441|[calc_checksum_by_commit] u=19 ret:0, checksum:1609128030, checksum_log_ts:9223372036854775807|[submit_log] u=43 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=710 trans_version:1659927274259081, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8314780, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=5 arg1:140085233547952|total_timeu=995) [2022-08-08 10:54:34.268992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:34.291234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:34.300085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.333592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.339328] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.341807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.343842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:34.350812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.353724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.361160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.371629] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.374482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.374526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.374571] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.376978] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:34.385044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.419234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.424105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.434557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.445196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.456370] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:34.459554] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:34.459609] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:34.478233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.478266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.494035] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.494107] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=62] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.494115] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.494184] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.494346] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.494414] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.494679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.495669] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D47-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.495876] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.496020] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.496138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.496294] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:34.498583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.498814] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.501935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:34.540241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:34.540390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.612759] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.617314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:34.617462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:34.693253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.704328] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.714864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:34.723912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.777089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:34.800120] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:34.884134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:34.993473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:35.088686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.088877] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FF-0-0] [lt=76] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.089282] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FF-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.089302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FF-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.089307] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FF-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.089458] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C47FF-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.106387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.108143] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:35.108202] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668028, retry interval=1000000, ret=-4179) [2022-08-08 10:54:35.125132] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:35.127286] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:35.127351] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668023, retry interval=1000000, ret=-4179) [2022-08-08 10:54:35.136904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:35.294612] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927275294583, cur_ts_=1659927275270297, delta=24286) [2022-08-08 10:54:35.296273] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28131, time_dist: generate_cluster_version=28128) [2022-08-08 10:54:35.437060] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:54:35.462269] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:35.465488] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:35.465525] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:35.623328] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:35.680049] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14273, time_dist: do_cluster_heartbeat=12, generate_cluster_version=14256) [2022-08-08 10:54:36.113150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:36.113279] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:36.134970] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:36.135017] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668027, retry interval=1000000, ret=-4179) [2022-08-08 10:54:36.150563] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:36.150805] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:36.152038] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:36.152100] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668022, retry interval=1000000, ret=-4179) [2022-08-08 10:54:36.468770] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:36.472301] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:36.472336] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:36.505165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:36.505325] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:36.505469] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:36.505816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:37.154127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4802-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.155090] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4802-0-0] [lt=920] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.155108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4802-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.155121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4802-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.155132] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4802-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.170860] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:37.170898] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668026, retry interval=1000000, ret=-4179) [2022-08-08 10:54:37.172700] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.172859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:37.174444] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:37.174504] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668021, retry interval=1000000, ret=-4179) [2022-08-08 10:54:37.299613] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659927277298712 2022-08-08 02:54:37.298712|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:11214903065894238151, inc:84086658, addr:{ip:"192.168.11.128", port:2882}, t:1659927277298000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5D4C-0-0, start_time:1659927277298588, snapshot:1659927277202422|[end_task] u=139 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927277298870|[calc_checksum_by_commit] u=20 ret:0, checksum:4188301117, checksum_log_ts:9223372036854775807|[submit_log] u=74 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=561 trans_version:1659927277298892, uref:1073741825|[exiting] u=23 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8300202, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=862) [2022-08-08 10:54:37.479623] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:37.482733] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:37.482774] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:38.180785] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:38.180956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:38.181142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:38.205474] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:38.205533] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668025, retry interval=1000000, ret=-4179) [2022-08-08 10:54:38.207353] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:38.208552] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:38.208627] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668020, retry interval=1000000, ret=-4179) [2022-08-08 10:54:38.278336] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13755) [2022-08-08 10:54:38.278533] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14651, time_dist: generate_cluster_version=14648) [2022-08-08 10:54:38.319934] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16112) [2022-08-08 10:54:38.321249] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18236, time_dist: generate_cluster_version=18008) [2022-08-08 10:54:38.485504] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:38.489165] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:38.489202] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:38.501751] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13876) [2022-08-08 10:54:38.502373] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15283, time_dist: generate_cluster_version=15280) [2022-08-08 10:54:38.515131] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:38.515154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:38.518882] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17090) [2022-08-08 10:54:38.519333] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23281, time_dist: generate_cluster_version=23278) [2022-08-08 10:54:38.730120] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13653) [2022-08-08 10:54:38.730330] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15901, time_dist: generate_cluster_version=15900) [2022-08-08 10:54:38.749881] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11877) [2022-08-08 10:54:38.750102] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12837, time_dist: generate_cluster_version=12834) [2022-08-08 10:54:39.184973] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927279184963, cur_ts_=1659927279151849, delta=33114) [2022-08-08 10:54:39.197561] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12380, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12366) [2022-08-08 10:54:39.222624] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:39.222686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:39.253790] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:39.255685] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=357] [dc=0] task process failed, start retry(max retry time=9223372036854668024, retry interval=1000000, ret=-4179) [2022-08-08 10:54:39.259428] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=164] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:39.259670] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] task process failed, start retry(max retry time=9223372036854668019, retry interval=1000000, ret=-4179) [2022-08-08 10:54:39.313987] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10925) [2022-08-08 10:54:39.314308] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12341, time_dist: do_cluster_heartbeat=6, generate_cluster_version=12333) [2022-08-08 10:54:39.343259] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29235) [2022-08-08 10:54:39.343491] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=41571, time_dist: do_cluster_heartbeat=25, generate_cluster_version=41544) [2022-08-08 10:54:39.507089] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:39.511944] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927279511934, cur_ts_=1659927279476063, delta=35871) [2022-08-08 10:54:39.525413] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:39.525451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:39.526655] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:39.526685] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:39.526695] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:39.526705] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:39.588044] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22626) [2022-08-08 10:54:39.590174] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25723, time_dist: generate_cluster_version=25718) [2022-08-08 10:54:39.596821] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33390, time_dist: generate_cluster_version=33386) [2022-08-08 10:54:39.664308] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:54:39.664341] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:54:40.015016] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14711168}) [2022-08-08 10:54:40.087386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.105325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.209139] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927278260438, end_time=1659927280209125, elapsed_time=1948687, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:54:40.270852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.290204] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:40.290254] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668023, retry interval=1000000, ret=-4179) [2022-08-08 10:54:40.291301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:40.292874] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:40.293137] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] task process failed, start retry(max retry time=9223372036854668018, retry interval=1000000, ret=-4179) [2022-08-08 10:54:40.429955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:40.469631] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:40.488951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.499373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.504272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.538958] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:40.543387] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=1172] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:40.543414] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:40.545944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.545959] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:40.572376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.587922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:40.587998] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:40.593125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.599920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.635694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:40.645944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.728361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.810434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.887039] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927280887022, cur_ts_=1659927280858595, delta=28427) [2022-08-08 10:54:40.934029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:40.978108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:40.996652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:41.317745] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:41.317793] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668022, retry interval=1000000, ret=-4179) [2022-08-08 10:54:41.321340] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:41.321372] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668017, retry interval=1000000, ret=-4179) [2022-08-08 10:54:41.545689] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:41.555287] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:41.555328] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:41.595199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:41.595329] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:41.595445] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:41.595536] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:41.595947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:41.648477] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] clock out of order(cur_ts=1659927281648472, cur_ts_=1659927281620761, delta=27711) [2022-08-08 10:54:42.050671] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927282050661, cur_ts_=1659927282016107, delta=34554) [2022-08-08 10:54:42.346819] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:42.346893] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668016, retry interval=1000000, ret=-4179) [2022-08-08 10:54:42.351553] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:42.351638] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854668021, retry interval=1000000, ret=-4179) [2022-08-08 10:54:42.558209] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:42.561406] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:42.561451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:42.641281] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] clock out of order(cur_ts=1659927282641272, cur_ts_=1659927282605030, delta=36242) [2022-08-08 10:54:42.721743] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14068) [2022-08-08 10:54:42.722520] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15838, time_dist: do_cluster_heartbeat=11, generate_cluster_version=15822) [2022-08-08 10:54:42.772539] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13674, time_dist: generate_cluster_version=13670) [2022-08-08 10:54:43.172918] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927283172912, cur_ts_=1659927283143974, delta=28938) [2022-08-08 10:54:43.196627] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24050) [2022-08-08 10:54:43.196929] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25203, time_dist: generate_cluster_version=25199) [2022-08-08 10:54:43.201009] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28681, time_dist: print_stat=6, do_cluster_heartbeat=3, generate_cluster_version=28615) [2022-08-08 10:54:43.337064] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927283337055, cur_ts_=1659927283314973, delta=22082) [2022-08-08 10:54:43.394003] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:43.394068] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668020, retry interval=1000000, ret=-4179) [2022-08-08 10:54:43.400419] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:43.400455] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668015, retry interval=1000000, ret=-4179) [2022-08-08 10:54:43.564110] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:43.567151] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:43.567188] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:43.758282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:43.866772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:43.912311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:43.937062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.085173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.116198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:44.117674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.123650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:44.131885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.137963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.147674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:44.222741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.263090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.271080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.271364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:44.276431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.282926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.293293] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:44.310059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:44.325059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.356075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.357393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.359914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.360062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:44.374600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.375458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.384465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.387304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.391576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.391981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.397634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.398939] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:44.409156] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.431293] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:44.431334] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668014, retry interval=1000000, ret=-4179) [2022-08-08 10:54:44.434256] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:44.434310] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668019, retry interval=1000000, ret=-4179) [2022-08-08 10:54:44.438223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.448434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.453428] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.469333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.497727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.503573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:44.513954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=204] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.518399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:44.521572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.559573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:44.559777] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.571336] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:44.574393] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:44.574432] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:44.637557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:44.641575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:44.648054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.709913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.727238] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.731703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:44.741451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.799638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:44.827616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:44.901463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:45.013424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:45.161490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:45.457659] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:45.457725] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668013, retry interval=1000000, ret=-4179) [2022-08-08 10:54:45.463246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:45.463443] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=179] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:45.467172] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:45.467441] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668018, retry interval=1000000, ret=-4179) [2022-08-08 10:54:45.577543] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:45.580699] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:45.580755] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:45.668953] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2D68-0-0] [lt=20] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927285665548 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927285666672 abs_expired_time=1659927287598274 stmt_timeout=1931602 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21589691 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927285610825 start_version=-1 trans_version=1659927285665538 commit_version=0 stmt_start_time=1659927285649310 abs_expired_time=1659927315563033 stmt_timeout=29913723 abs_lock_wait_timeout=1659927315563033 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:5020481132122142088, inc:84086966, addr:"192.168.11.128:2882", t:1659927285649310} checksum_log_ts=9223372036854775807, version=1659927285665548) [2022-08-08 10:54:45.682741] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:46.305891] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927286305853, cur_ts_=1659927286283133, delta=22720) [2022-08-08 10:54:46.494063] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:46.494105] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668017, retry interval=1000000, ret=-4179) [2022-08-08 10:54:46.496932] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:46.497001] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668012, retry interval=1000000, ret=-4179) [2022-08-08 10:54:46.583399] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:46.587707] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:46.587743] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:46.689093] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:46.689200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:46.689216] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:46.689226] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:47.438707] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1869] [dc=0] clock out of order(cur_ts=1659927287438699, cur_ts_=1659927287414070, delta=24629) [2022-08-08 10:54:47.497076] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4811-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:47.498543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4811-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:47.527637] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:47.527677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:47.528785] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:47.528818] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668011, retry interval=1000000, ret=-4179) [2022-08-08 10:54:47.530662] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:47.530697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:47.532496] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:47.532644] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854668016, retry interval=1000000, ret=-4179) [2022-08-08 10:54:47.593161] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:47.596386] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:47.596424] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:47.700481] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D6E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:47.700499] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D6E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:47.700504] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D6E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:47.700508] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D6E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:47.700574] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D6E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:48.556037] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:48.556148] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:48.557871] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:48.558138] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=230] [dc=0] task process failed, start retry(max retry time=9223372036854668010, retry interval=1000000, ret=-4179) [2022-08-08 10:54:48.559830] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:48.559866] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668015, retry interval=1000000, ret=-4179) [2022-08-08 10:54:48.601232] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:48.604872] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:48.604962] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:48.673074] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927288673069, cur_ts_=1659927288643659, delta=29410) [2022-08-08 10:54:48.936428] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927288936418, cur_ts_=1659927288909493, delta=26925) [2022-08-08 10:54:49.560139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4853-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:49.560299] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4853-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:49.576679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:49.587416] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:49.587469] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668014, retry interval=1000000, ret=-4179) [2022-08-08 10:54:49.598035] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:49.598069] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668009, retry interval=1000000, ret=-4179) [2022-08-08 10:54:49.606308] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18133) [2022-08-08 10:54:49.607600] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:49.610639] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:49.610664] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:49.667107] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:54:49.667137] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:54:49.726834] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927289726825, cur_ts_=1659927289698455, delta=28370) [2022-08-08 10:54:49.947937] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14715488}) [2022-08-08 10:54:49.996046] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15940, time_dist: generate_cluster_version=15936) [2022-08-08 10:54:50.107819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.124212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.137711] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927288262243, end_time=1659927290137700, elapsed_time=1875457, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:54:50.286164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=801] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.451522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:50.491549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:50.513797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.515894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.521100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.567920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.569673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:50.595909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.607714] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:50.609020] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:50.609073] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854668013, retry interval=1000000, ret=-4179) [2022-08-08 10:54:50.613272] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:50.613713] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.617516] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:50.617553] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:50.623679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:50.624540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.624885] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:50.624922] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668008, retry interval=1000000, ret=-4179) [2022-08-08 10:54:50.652477] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:50.662850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.753176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:50.793612] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=293] [dc=0] clock out of order(cur_ts=1659927290793604, cur_ts_=1659927290766835, delta=26769) [2022-08-08 10:54:50.825980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.954893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:50.995912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:51.017104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:51.610839] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4817-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:51.621896] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:51.626866] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4856-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:51.627980] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:51.629455] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:51.629487] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668012, retry interval=1000000, ret=-4179) [2022-08-08 10:54:51.638388] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:51.638414] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:51.641096] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11940, time_dist: generate_cluster_version=11937) [2022-08-08 10:54:51.649468] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:51.650778] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:51.650873] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] task process failed, start retry(max retry time=9223372036854668007, retry interval=1000000, ret=-4179) [2022-08-08 10:54:52.631211] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6021-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:52.640820] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:52.644789] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:52.644814] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:52.648042] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:52.650512] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:52.651495] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:52.651527] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668011, retry interval=1000000, ret=-4179) [2022-08-08 10:54:52.673411] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:52.674905] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:52.674968] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668006, retry interval=1000000, ret=-4179) [2022-08-08 10:54:52.771190] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927292771180, cur_ts_=1659927292748050, delta=23130) [2022-08-08 10:54:53.425114] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] clock out of order(cur_ts=1659927293425108, cur_ts_=1659927293397993, delta=27115) [2022-08-08 10:54:53.650309] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:53.653644] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:53.653668] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:53.671001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:53.672373] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:53.672424] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668010, retry interval=1000000, ret=-4179) [2022-08-08 10:54:53.680064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C481A-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:53.698153] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:53.699906] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:53.700380] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854668005, retry interval=1000000, ret=-4179) [2022-08-08 10:54:53.776523] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:53.883561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:53.928604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:53.958997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.101753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.133018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:54.138963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:54.142368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.156209] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.163512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:54.164280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.242013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.280767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.291676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.293215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.295089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:54.306658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.309143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:54.329802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:54.343226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.373968] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.382178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.383900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:54:54.383913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.392382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.396786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.402867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.409861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.409914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.415614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:54:54.416757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.417089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.425584] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.454879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.464236] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.469705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.489542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.521067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:54.521560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.531759] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.539193] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:54:54.539304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.580931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.583792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:54.653257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:54.656274] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:54.659868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:54:54.660006] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:54.660040] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:54.666639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.674219] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C499C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:54.700538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:54.702171] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:54.702270] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854668009, retry interval=1000000, ret=-4179) [2022-08-08 10:54:54.702361] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6024-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:54.718223] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:54.719468] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:54.719504] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668004, retry interval=1000000, ret=-4179) [2022-08-08 10:54:54.733681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.744001] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.752064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:54.763341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.816903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:54:54.846226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:54.918364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:54:55.033788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:55.186014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:54:55.308907] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:54:55.463283] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:54:55.665574] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:55.669536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:55.669574] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:55.704535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C485C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:55.728471] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:55.729646] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:55.729679] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668008, retry interval=1000000, ret=-4179) [2022-08-08 10:54:55.746986] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:55.747118] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] task process failed, start retry(max retry time=9223372036854668003, retry interval=1000000, ret=-4179) [2022-08-08 10:54:55.754019] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:56.671788] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:56.675264] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:56.675304] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:56.809493] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:56.809538] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668002, retry interval=1000000, ret=-4179) [2022-08-08 10:54:56.821964] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11387) [2022-08-08 10:54:56.822437] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:56.822500] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668007, retry interval=1000000, ret=-4179) [2022-08-08 10:54:57.681290] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:57.684350] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:57.684391] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:57.824286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4820-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:57.824539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4820-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:57.829945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:57.830143] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:57.831832] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:57.831932] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854668001, retry interval=1000000, ret=-4179) [2022-08-08 10:54:57.848306] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:57.851724] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:57.851783] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668006, retry interval=1000000, ret=-4179) [2022-08-08 10:54:58.687223] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:58.691953] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:58.691978] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:58.751416] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D8F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.751476] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D8F-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.751596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D8F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.753280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.753302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.753312] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.753354] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:54:58.807559] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927298807550, cur_ts_=1659927298782012, delta=25538) [2022-08-08 10:54:58.853572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49A2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:58.853709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49A2-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:54:58.858219] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:58.858336] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854668000, retry interval=1000000, ret=-4179) [2022-08-08 10:54:58.874163] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:58.874194] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668005, retry interval=1000000, ret=-4179) [2022-08-08 10:54:59.322504] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13017) [2022-08-08 10:54:59.322721] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13914, time_dist: generate_cluster_version=13912) [2022-08-08 10:54:59.670299] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:54:59.670338] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:54:59.700581] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:54:59.706375] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:54:59.706432] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:54:59.708628] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11357, time_dist: do_cluster_heartbeat=12, generate_cluster_version=11340) [2022-08-08 10:54:59.736112] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] clock out of order(cur_ts=1659927299736103, cur_ts_=1659927299703918, delta=32185) [2022-08-08 10:54:59.874943] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14719840}) [2022-08-08 10:54:59.903237] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:59.903271] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667999, retry interval=1000000, ret=-4179) [2022-08-08 10:54:59.921052] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:54:59.921088] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854668004, retry interval=1000000, ret=-4179) [2022-08-08 10:55:00.080548] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927298263119, end_time=1659927300080535, elapsed_time=1817416, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:55:00.132522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.141554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.309292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.467461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:00.513600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:00.534421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:00.536363] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:00.544877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.585138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:00.588954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.612747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:00.638571] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:00.641010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.671742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:00.681785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:00.709499] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:00.712675] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:00.712751] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:00.774159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:00.850724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:00.936823] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:00.936866] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667998, retry interval=1000000, ret=-4179) [2022-08-08 10:55:00.947850] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:00.947914] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854668003, retry interval=1000000, ret=-4179) [2022-08-08 10:55:00.979795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:01.012982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:01.041829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:01.716819] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:01.720113] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:01.720149] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:01.979911] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:01.980861] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667997, retry interval=1000000, ret=-4179) [2022-08-08 10:55:02.016099] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:02.016137] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854668002, retry interval=1000000, ret=-4179) [2022-08-08 10:55:02.723434] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:02.728725] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=334] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:02.728763] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:02.773287] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D9B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:02.773461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D9B-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:02.773503] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D9B-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:02.773653] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2D9B-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:03.028443] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49A8-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:03.028486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49A8-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:03.044842] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:03.044862] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:03.046608] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:03.046672] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667996, retry interval=1000000, ret=-4179) [2022-08-08 10:55:03.054921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:03.055159] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:03.057907] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:03.058108] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] task process failed, start retry(max retry time=9223372036854668001, retry interval=1000000, ret=-4179) [2022-08-08 10:55:03.122518] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19305) [2022-08-08 10:55:03.122832] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25971, time_dist: generate_cluster_version=25963, cluster_service_self_check=5) [2022-08-08 10:55:03.656717] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927303656706, cur_ts_=1659927303628457, delta=28249) [2022-08-08 10:55:03.737542] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:03.746946] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:03.747526] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=564] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:03.801918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:03.908007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:03.946211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:03.975068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.055807] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4868-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:04.055851] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4868-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:04.083715] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:04.083764] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667995, retry interval=1000000, ret=-4179) [2022-08-08 10:55:04.106611] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:04.106745] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854668000, retry interval=1000000, ret=-4179) [2022-08-08 10:55:04.121035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.142329] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14642) [2022-08-08 10:55:04.152167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:04.153201] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26229, time_dist: generate_cluster_version=26225) [2022-08-08 10:55:04.164077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:04.165826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.172358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.182507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.188797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=226] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:04.262322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.297059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.314387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:04.314938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.316921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.325322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.325322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:04.355151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:04.359547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.391937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.401240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:04.402226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.407647] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.408627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.413173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.423123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.428577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.432485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.432687] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.433511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.436874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:04.444345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.473595] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.487993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.496111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.505036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.542073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=379] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:04.546472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.556557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.558546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.562395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:04.600974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.601859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:04.674091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:04.675121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:04.685670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.749453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.752371] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=299] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:04.757964] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:04.758001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:04.759389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.769943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:04.780188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.781997] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DA1-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:04.783004] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:04.839829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:04.863524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:04.938860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:05.056912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:05.110221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49AB-0-0] [lt=82] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:05.115352] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=200] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:05.115565] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=157] [dc=0] task process failed, start retry(max retry time=9223372036854667994, retry interval=1000000, ret=-4179) [2022-08-08 10:55:05.116208] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11428, time_dist: do_cluster_heartbeat=8, generate_cluster_version=11418) [2022-08-08 10:55:05.132980] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:05.133020] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667999, retry interval=1000000, ret=-4179) [2022-08-08 10:55:05.133409] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=8199] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17004) [2022-08-08 10:55:05.204737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:05.760784] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:05.764523] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:05.764573] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:05.824345] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:06.137077] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C482C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:06.144439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:06.148117] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:06.148166] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667993, retry interval=1000000, ret=-4179) [2022-08-08 10:55:06.160142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:06.161838] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:06.161902] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667998, retry interval=1000000, ret=-4179) [2022-08-08 10:55:06.767611] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:06.772650] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:06.772678] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:06.792347] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DA7-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:06.792391] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DA7-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:06.792398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DA7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:06.793763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:06.793944] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:06.793992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:07.168506] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927307168497, cur_ts_=1659927307146268, delta=22229) [2022-08-08 10:55:07.202177] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=275] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:07.202237] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667997, retry interval=1000000, ret=-4179) [2022-08-08 10:55:07.214970] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24847, time_dist: generate_cluster_version=24842) [2022-08-08 10:55:07.217333] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:07.217404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667992, retry interval=1000000, ret=-4179) [2022-08-08 10:55:07.775643] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:07.780531] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:07.780745] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=196] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:07.799730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DAA-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:07.799751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DAA-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:07.800013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DAA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:08.204791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.204857] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.204957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.205073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.205215] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.205373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.205551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.205824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.206012] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.206154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.206343] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.206440] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C486E-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.221127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C482F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.221151] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C482F-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.221162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C482F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.221201] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C482F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.235312] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:08.236764] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:08.236812] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667996, retry interval=1000000, ret=-4179) [2022-08-08 10:55:08.247877] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:08.247916] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667991, retry interval=1000000, ret=-4179) [2022-08-08 10:55:08.528138] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=18743) [2022-08-08 10:55:08.529386] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20650, time_dist: generate_cluster_version=20646) [2022-08-08 10:55:08.784028] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:08.792274] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:08.792703] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=414] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:08.807000] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DAD-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:08.807042] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DAD-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:08.807384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:08.807445] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:08.807570] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:09.017192] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927309017188, cur_ts_=1659927308984395, delta=32793) [2022-08-08 10:55:09.275638] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:09.275672] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667990, retry interval=1000000, ret=-4179) [2022-08-08 10:55:09.284096] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:09.284379] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:09.285320] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:09.287110] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=1728] [dc=0] task process failed, start retry(max retry time=9223372036854667995, retry interval=1000000, ret=-4179) [2022-08-08 10:55:09.666243] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:55:09.666281] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:55:09.751728] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=817] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11071, time_dist: do_cluster_heartbeat=11, generate_cluster_version=11055) [2022-08-08 10:55:09.752503] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11057, time_dist: do_cluster_heartbeat=6, generate_cluster_version=11048) [2022-08-08 10:55:09.794878] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:09.797921] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:09.797945] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:09.848239] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14724192}) [2022-08-08 10:55:09.853767] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11105) [2022-08-08 10:55:10.033170] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=2283] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927308264718, end_time=1659927310033159, elapsed_time=1768441, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:55:10.156222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.157087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.309249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:10.309558] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=293] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:10.316809] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:10.316849] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667994, retry interval=1000000, ret=-4179) [2022-08-08 10:55:10.321340] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:10.321381] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667989, retry interval=1000000, ret=-4179) [2022-08-08 10:55:10.324446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.484831] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:10.533622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:10.557940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:10.560715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:10.565270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.603063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:10.609045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.637832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:10.655146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:10.658758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.693474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:10.703987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:10.799165] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:10.800961] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:10.806157] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:10.806204] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:10.817747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DB3-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:10.817802] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DB3-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:10.817984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:10.818088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:10.818271] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:10.818338] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:10.867998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:10.998911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:11.032704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:11.062870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:11.319111] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.319261] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.319542] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=64] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.320555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=234] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.320879] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.320984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.320997] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:11.344328] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:11.344367] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667993, retry interval=1000000, ret=-4179) [2022-08-08 10:55:11.356632] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:11.356801] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667988, retry interval=1000000, ret=-4179) [2022-08-08 10:55:11.808604] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:11.812182] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:11.812219] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:12.358064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4835-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:12.358090] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4835-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:12.358215] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4835-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:12.358266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4835-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:12.397975] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:12.398234] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:12.398665] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13713) [2022-08-08 10:55:12.410567] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:12.410607] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667987, retry interval=1000000, ret=-4179) [2022-08-08 10:55:12.415450] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31712, time_dist: do_cluster_heartbeat=12, generate_cluster_version=31696) [2022-08-08 10:55:12.416041] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17324) [2022-08-08 10:55:12.417349] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28242, time_dist: generate_cluster_version=28240) [2022-08-08 10:55:12.426700] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=172] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:12.428781] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] task process failed, start retry(max retry time=9223372036854667992, retry interval=1000000, ret=-4179) [2022-08-08 10:55:12.648365] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:12.814987] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:12.818197] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:12.818240] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:12.830550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:12.830851] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:12.831093] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:13.356937] WARN [SQL] execute_get_plan (ob_sql.cpp:3081) [11024][0][YB42C0A80B80-0005E3CF9BC835F1-0-0] [lt=34] [dc=0] fail to get plan retry(ret=-5138) [2022-08-08 10:55:13.387214] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835F1-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:55:13.389696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835F1-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:55:13.389776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835F1-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:55:13.423945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.424276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.424321] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.424349] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.424361] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.424371] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.424614] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC603F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.429215] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.477949] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927313477945, cur_ts_=1659927313445180, delta=32765) [2022-08-08 10:55:13.497852] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.497873] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.498014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.498154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:13.499984] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=327] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:13.500039] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667991, retry interval=1000000, ret=-4179) [2022-08-08 10:55:13.500529] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:13.500550] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667986, retry interval=1000000, ret=-4179) [2022-08-08 10:55:13.817471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:13.822421] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:13.825686] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:13.825732] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:13.929221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:13.971421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:13.992554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.137080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.171087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:14.187953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.188996] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=272] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.189233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=1661] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:14.201005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.214234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=607] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:14.282572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.313647] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.329784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:14.331768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.339760] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.344868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:14.344871] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.373975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:14.377416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.409191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.417694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.417993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:14.429154] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.430133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.432475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.440092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.449405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.450609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.450688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.455255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:14.455299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.461159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.491895] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.503126] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.513710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.526397] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:14.526529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:14.530545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.534989] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:14.535043] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667990, retry interval=1000000, ret=-4179) [2022-08-08 10:55:14.543146] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:14.543391] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=232] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:14.544912] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:14.544983] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667985, retry interval=1000000, ret=-4179) [2022-08-08 10:55:14.566891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:14.571930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.574340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.582388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:14.582429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.617528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.621012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:14.692027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:14.692101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:14.702478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.768266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.775121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.787128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:14.799112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.828401] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:14.831471] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:14.831507] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:14.861575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:14.884323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:14.959859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:15.074977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:15.224731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:15.499058] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927315499050, cur_ts_=1659927315454244, delta=44806) [2022-08-08 10:55:15.510342] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:55:15.547218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49BA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:15.547436] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49BA-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:15.558892] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10579, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10569) [2022-08-08 10:55:15.575106] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:15.575156] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667984, retry interval=1000000, ret=-4179) [2022-08-08 10:55:15.579288] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:15.579340] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667989, retry interval=1000000, ret=-4179) [2022-08-08 10:55:15.834173] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:15.837489] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:15.837530] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:15.856183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:15.856534] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:15.856555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:15.890457] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:16.587848] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C483B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.587902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C483B-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.587918] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C483B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.587932] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C483B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.587956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C483B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.607688] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:16.607832] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] task process failed, start retry(max retry time=9223372036854667988, retry interval=1000000, ret=-4179) [2022-08-08 10:55:16.609450] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.609470] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:16.610692] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:16.610727] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667983, retry interval=1000000, ret=-4179) [2022-08-08 10:55:16.843102] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:16.846972] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:16.847014] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:16.868744] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DC5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.868805] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DC5-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.869529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DC5-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.869713] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DC5-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.870507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.870554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.870572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:16.870589] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:17.496376] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927317496372, cur_ts_=1659927317459685, delta=36687) [2022-08-08 10:55:17.657303] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:17.657358] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667982, retry interval=1000000, ret=-4179) [2022-08-08 10:55:17.667265] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:17.667434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=150] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:17.669491] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:17.669545] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667987, retry interval=1000000, ret=-4179) [2022-08-08 10:55:17.851889] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:17.855036] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:17.855072] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:18.671996] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C487D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.672162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C487D-0-0] [lt=134] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.672177] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C487D-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.672182] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C487D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.672211] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C487D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.697280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.698172] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.704419] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:18.704571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667981, retry interval=1000000, ret=-4179) [2022-08-08 10:55:18.707616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.707708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:18.708855] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:18.708956] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] task process failed, start retry(max retry time=9223372036854667986, retry interval=1000000, ret=-4179) [2022-08-08 10:55:18.857761] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:18.860894] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:18.860918] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:19.665305] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:55:19.665337] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:55:19.708526] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6048-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:19.708616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6048-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:19.708632] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6048-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:19.708746] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6048-0-0] [lt=99] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:19.708772] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6048-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:19.739629] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:19.739687] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667980, retry interval=1000000, ret=-4179) [2022-08-08 10:55:19.752154] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:19.752206] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667985, retry interval=1000000, ret=-4179) [2022-08-08 10:55:19.798210] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22841) [2022-08-08 10:55:19.862678] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:19.865416] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:19.865452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:19.904783] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14728512}) [2022-08-08 10:55:19.952702] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927319952695, cur_ts_=1659927319928844, delta=23851) [2022-08-08 10:55:19.953982] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34567) [2022-08-08 10:55:19.974146] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18998) [2022-08-08 10:55:20.056388] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=150] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=50305) [2022-08-08 10:55:20.058362] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18610, time_dist: generate_cluster_version=18607) [2022-08-08 10:55:20.061521] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=56441, time_dist: do_cluster_heartbeat=10, generate_cluster_version=56406) [2022-08-08 10:55:20.147383] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=248] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927318265576, end_time=1659927320147368, elapsed_time=1881792, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:55:20.176600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.178942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.344922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.510283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:20.551755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:20.577014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:20.584872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.585046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:20.618694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:20.628495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.655327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:20.670811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:20.684117] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.711867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:20.722250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:20.744210] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4843-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:20.781368] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:20.781404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667979, retry interval=1000000, ret=-4179) [2022-08-08 10:55:20.792285] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:20.792323] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667984, retry interval=1000000, ret=-4179) [2022-08-08 10:55:20.821478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:20.870076] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:20.873242] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:20.873302] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:20.883453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:20.922756] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:20.923146] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:21.023065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:21.054088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:21.084863] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:21.401925] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15888, time_dist: generate_cluster_version=15883) [2022-08-08 10:55:21.401984] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15904, time_dist: generate_cluster_version=15902) [2022-08-08 10:55:21.783916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784043] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=104] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784370] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784481] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=92] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.784894] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=285] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.785001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604B-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:21.813718] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:21.813837] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667983, retry interval=1000000, ret=-4179) [2022-08-08 10:55:21.815955] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:21.816020] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667978, retry interval=1000000, ret=-4179) [2022-08-08 10:55:21.875960] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:21.879586] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:21.879651] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:21.930293] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:22.846448] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:22.846580] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] task process failed, start retry(max retry time=9223372036854667982, retry interval=1000000, ret=-4179) [2022-08-08 10:55:22.855198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:22.855224] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:22.857190] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:22.857280] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667977, retry interval=1000000, ret=-4179) [2022-08-08 10:55:22.884339] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:22.890416] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:22.890459] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:23.837164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:23.859045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:23.860562] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604E-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:23.860560] WARN [COMMON] on_full_return (ob_io_struct.cpp:2546) [10864][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] push io request into callback queue failed(ret=-4072, req={is_inited_:true, is_finished_:true, is_canceled_:true, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:492}, offset_:991232, size_:20480, flag_:{mode_:0, category_:1, wait_event_id_:1, reserved_:0}, write_buf_:null, callback_:0x7f6969a99610}, deadline_ts_:1659927323858753, control_block_:0x7f6833bc4a40, raw_buf_:null, io_buf_:0x7f67cc233000, io_offset_:991232, io_size_:20480, complete_size_:20480, time_log_:{begin_ts_:1659927323858752, enqueue_ts_:1659927323858753, dequeue_ts_:1659927323859017, submit_ts_:1659927323859020, return_ts_:1659927323860533, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1659927323860535}, channel_:0x7f69a4ef6310, ref_cnt_:2, out_ref_cnt_:0, trace_id_:YB42C0A80B80-0005E3CFC5EC604E-0-0, ret_code_:{io_ret_:-4072, fs_errno_:0}, retry_count_:0, copied_callback_:0x7f67e303b0b0}) [2022-08-08 10:55:23.860678] WARN [COMMON] get_events (ob_io_struct.cpp:2500) [10864][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] process full return io request failed(ret=-4072, *req={is_inited_:true, is_finished_:true, is_canceled_:true, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:492}, offset_:991232, size_:20480, flag_:{mode_:0, category_:1, wait_event_id_:1, reserved_:0}, write_buf_:null, callback_:0x7f6969a99610}, deadline_ts_:1659927323858753, control_block_:0x7f6833bc4a40, raw_buf_:null, io_buf_:0x7f67cc233000, io_offset_:991232, io_size_:20480, complete_size_:20480, time_log_:{begin_ts_:1659927323858752, enqueue_ts_:1659927323858753, dequeue_ts_:1659927323859017, submit_ts_:1659927323859020, return_ts_:1659927323860533, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1659927323860535}, channel_:0x7f69a4ef6310, ref_cnt_:1, out_ref_cnt_:0, trace_id_:YB42C0A80B80-0005E3CFC5EC604E-0-0, ret_code_:{io_ret_:-4072, fs_errno_:0}, retry_count_:0, copied_callback_:0x7f67e303b0b0}) [2022-08-08 10:55:23.860707] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC604E-0-0] [lt=83] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:23.868045] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:23.868104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667981, retry interval=1000000, ret=-4179) [2022-08-08 10:55:23.879148] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:23.879200] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667976, retry interval=1000000, ret=-4179) [2022-08-08 10:55:23.900105] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:23.903432] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:23.903470] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:23.949358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:23.955960] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DDA-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:23.957493] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DDA-0-0] [lt=1503] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:23.958206] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DDA-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:23.958352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DDA-0-0] [lt=95] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:23.958528] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DDA-0-0] [lt=160] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:23.993504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.014373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.159112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.186343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:24.204234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.210519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.214606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:24.225025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.236947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:24.307912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.343778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.345689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:24.354961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.355040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.363379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.366499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:24.395180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:24.396013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.427922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.438276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.439470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:24.448229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.449752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=737] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.454156] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.460221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.465714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.469552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.471521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.472729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:24.474247] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.477947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.512357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.519104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.533548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.553994] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.582151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:24.591125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.600169] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.600372] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.603128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:24.641683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.644731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:24.709192] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:24.716538] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:24.726464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.790820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.792436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.805701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:24.819882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.870388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.870844] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.871019] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=142] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.871065] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.871303] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.871569] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=48] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.871812] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4849-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.879707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:24.883620] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4886-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.883827] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4886-0-0] [lt=154] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.892432] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.894413] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:24.894446] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667980, retry interval=1000000, ret=-4179) [2022-08-08 10:55:24.901580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:24.903517] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:24.904883] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:24.904943] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667975, retry interval=1000000, ret=-4179) [2022-08-08 10:55:24.905998] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:24.909151] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:24.909185] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:24.978437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:25.097970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:25.244277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:25.911585] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:25.924154] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:25.925700] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=1528] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:25.988321] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE0-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.988620] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE0-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.988629] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE0-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.988634] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE0-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.988649] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE0-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.988661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE0-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.994343] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:25.994383] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667974, retry interval=1000000, ret=-4179) [2022-08-08 10:55:25.996478] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.997101] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=574] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.997121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:25.997141] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:26.014417] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:26.014446] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667979, retry interval=1000000, ret=-4179) [2022-08-08 10:55:26.018403] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24303) [2022-08-08 10:55:26.023318] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:26.028240] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38821, time_dist: print_stat=5, generate_cluster_version=38812) [2022-08-08 10:55:26.029241] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=541] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10264) [2022-08-08 10:55:26.041113] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10189) [2022-08-08 10:55:26.101567] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927326101561, cur_ts_=1659927326081232, delta=20329) [2022-08-08 10:55:26.174310] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18001) [2022-08-08 10:55:26.174515] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19160, time_dist: generate_cluster_version=19157) [2022-08-08 10:55:26.935836] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:26.939481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:26.939539] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:26.998200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4889-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:26.998338] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4889-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:26.998395] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4889-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:26.998657] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4889-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:26.998888] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4889-0-0] [lt=207] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:27.012788] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10475, time_dist: print_stat=5, generate_cluster_version=10467) [2022-08-08 10:55:27.041160] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:27.041198] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667978, retry interval=1000000, ret=-4179) [2022-08-08 10:55:27.042014] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:27.042041] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667973, retry interval=1000000, ret=-4179) [2022-08-08 10:55:27.943075] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:27.946575] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:27.946633] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:28.080961] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:28.081014] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667977, retry interval=1000000, ret=-4179) [2022-08-08 10:55:28.088634] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:28.088639] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15177) [2022-08-08 10:55:28.088671] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667972, retry interval=1000000, ret=-4179) [2022-08-08 10:55:28.643069] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19419, time_dist: generate_cluster_version=19416) [2022-08-08 10:55:28.949447] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:28.952109] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927328952100, cur_ts_=1659927328930657, delta=21443) [2022-08-08 10:55:29.005089] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:29.005174] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:29.068144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE9-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.068175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE9-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.068580] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.068602] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2DE9-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.068756] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.069340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.069660] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.069727] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:29.083837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C484F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:29.091105] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C488C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:29.091146] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C488C-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:29.112194] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:29.112255] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667971, retry interval=1000000, ret=-4179) [2022-08-08 10:55:29.113734] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:29.113763] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667976, retry interval=1000000, ret=-4179) [2022-08-08 10:55:29.667690] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:55:29.667709] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:55:29.760536] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927329760527, cur_ts_=1659927329734167, delta=26360) [2022-08-08 10:55:29.969401] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14732896}) [2022-08-08 10:55:30.012678] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:30.026021] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927330026014, cur_ts_=1659927330004905, delta=21109) [2022-08-08 10:55:30.031465] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:30.031493] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:30.033899] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20509, time_dist: generate_cluster_version=20505) [2022-08-08 10:55:30.149695] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:30.149744] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667970, retry interval=1000000, ret=-4179) [2022-08-08 10:55:30.151407] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:30.151443] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667975, retry interval=1000000, ret=-4179) [2022-08-08 10:55:30.168425] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927328266670, end_time=1659927330168411, elapsed_time=1901741, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:55:30.193434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.204182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.360992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.534610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:30.571294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:30.592157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:30.601529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:30.606021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.643263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:30.653615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.674961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:30.691588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:30.701756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.734031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:30.744099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:30.837979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:30.900210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:31.033546] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:31.037699] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:31.037739] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:31.073059] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2002] [dc=0] clock out of order(cur_ts=1659927331073050, cur_ts_=1659927331042514, delta=30536) [2022-08-08 10:55:31.073067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:31.073519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:31.075281] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34525, time_dist: generate_cluster_version=34522) [2022-08-08 10:55:31.104028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:31.107648] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:31.200989] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:31.201040] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667974, retry interval=1000000, ret=-4179) [2022-08-08 10:55:31.209066] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:31.209127] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667969, retry interval=1000000, ret=-4179) [2022-08-08 10:55:32.040938] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:32.046182] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:32.046207] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:32.257776] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:32.257812] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667968, retry interval=1000000, ret=-4179) [2022-08-08 10:55:32.259198] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:32.259231] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667973, retry interval=1000000, ret=-4179) [2022-08-08 10:55:32.269347] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20294, time_dist: generate_cluster_version=20292) [2022-08-08 10:55:32.648928] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:33.049273] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:33.056173] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:33.056227] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:33.261279] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4855-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:33.261539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4892-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:33.309291] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:33.309309] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:33.311083] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:33.311410] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:33.311566] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:33.315575] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3569] [dc=0] task process failed, start retry(max retry time=9223372036854667972, retry interval=1000000, ret=-4179) [2022-08-08 10:55:33.317645] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:33.317688] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667967, retry interval=1000000, ret=-4179) [2022-08-08 10:55:33.853775] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:33.968222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.016362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.031738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.042649] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10270, time_dist: do_cluster_heartbeat=9, generate_cluster_version=10257) [2022-08-08 10:55:34.059496] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:34.062592] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:34.062647] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:34.130124] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2DF8-0-0] [lt=29] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927334129430 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927334129420 abs_expired_time=1659927336059472 stmt_timeout=1930052 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21590361 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927334108658 start_version=-1 trans_version=1659927334129420 commit_version=0 stmt_start_time=1659927334124545 abs_expired_time=1659927364025725 stmt_timeout=29901180 abs_lock_wait_timeout=1659927364025725 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:9676235106290163331, inc:84088823, addr:"192.168.11.128:2882", t:1659927334124545} checksum_log_ts=9223372036854775807, version=1659927334129430) [2022-08-08 10:55:34.176098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.209766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:34.220747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.231546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.238134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:34.248529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.253885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:34.319403] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC605E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.320025] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC605E-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.320050] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC605E-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.320056] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC605E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.320509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC605E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.320531] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC605E-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.324360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.343520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.343539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.345008] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:34.345044] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667966, retry interval=1000000, ret=-4179) [2022-08-08 10:55:34.353289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:34.354840] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:34.354875] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667971, retry interval=1000000, ret=-4179) [2022-08-08 10:55:34.368536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.368645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:34.382363] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.382827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.388478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.388541] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:34.415081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:34.415294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.445951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.456583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.457898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:34.465559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.469161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.477108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.478705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.489052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.489131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.489223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:34.496482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.498016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.499655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.529281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.539954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.550041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.575821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.598903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:34.612555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.622495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.624470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.624519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:34.663306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.663339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:34.724866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:34.734093] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:34.745971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.807459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.812856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.821035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:34.838690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.901788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=128] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:34.924970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:34.995173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:35.065383] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:35.068514] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:35.068552] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:35.123432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:35.263224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:35.347269] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4895-0-0] [lt=70] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:35.375657] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:35.375832] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667965, retry interval=1000000, ret=-4179) [2022-08-08 10:55:35.377041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:35.378536] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:35.378602] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667970, retry interval=1000000, ret=-4179) [2022-08-08 10:55:35.533798] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:55:36.072224] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:36.075925] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:36.075963] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:36.117054] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:36.148766] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2DFE-0-0] [lt=147] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927336146699 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927336147801 abs_expired_time=1659927338078047 stmt_timeout=1930246 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21590395 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927336126456 start_version=-1 trans_version=1659927336146689 commit_version=0 stmt_start_time=1659927336140660 abs_expired_time=1659927366040995 stmt_timeout=29900335 abs_lock_wait_timeout=1659927366040995 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:16947611794440827541, inc:84088908, addr:"192.168.11.128:2882", t:1659927336140660} checksum_log_ts=9223372036854775807, version=1659927336146699) [2022-08-08 10:55:36.378749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6061-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:36.378798] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6061-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:36.378818] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6061-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:36.379009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6061-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:36.403087] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:36.403337] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667969, retry interval=1000000, ret=-4179) [2022-08-08 10:55:36.410259] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:36.410408] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:36.411687] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:36.411749] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667964, retry interval=1000000, ret=-4179) [2022-08-08 10:55:37.078506] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:37.081498] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:37.081533] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:37.153586] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E01-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:37.153672] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E01-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:37.153882] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:37.154076] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:37.154237] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:37.154323] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:37.405724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4898-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:37.433069] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:37.433155] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667968, retry interval=1000000, ret=-4179) [2022-08-08 10:55:37.435622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:37.437164] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:37.437223] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667963, retry interval=1000000, ret=-4179) [2022-08-08 10:55:37.929925] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927337929915, cur_ts_=1659927337893474, delta=36441) [2022-08-08 10:55:38.083985] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:38.090140] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:38.090164] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:38.441200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.441235] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.441373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.441403] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.441589] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.441664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.441878] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6064-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.466721] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:38.466784] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667962, retry interval=1000000, ret=-4179) [2022-08-08 10:55:38.468119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:38.469482] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:38.469544] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667967, retry interval=1000000, ret=-4179) [2022-08-08 10:55:39.043064] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927339043056, cur_ts_=1659927339016857, delta=26199) [2022-08-08 10:55:39.092305] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:39.095486] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:39.095525] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:39.437898] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927339437889, cur_ts_=1659927339416255, delta=21634) [2022-08-08 10:55:39.485651] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:39.486291] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:39.486406] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667961, retry interval=1000000, ret=-4179) [2022-08-08 10:55:39.501490] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:39.501778] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=175] [dc=0] task process failed, start retry(max retry time=9223372036854667966, retry interval=1000000, ret=-4179) [2022-08-08 10:55:39.665020] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:55:39.665046] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:55:39.895175] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14737216}) [2022-08-08 10:55:39.897576] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12862, time_dist: generate_cluster_version=12858) [2022-08-08 10:55:40.084762] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927338268039, end_time=1659927340084749, elapsed_time=1816710, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:55:40.097231] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:40.104877] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:40.104930] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:40.216918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.229123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.382984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.508932] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:40.508970] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667960, retry interval=1000000, ret=-4179) [2022-08-08 10:55:40.523168] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13451) [2022-08-08 10:55:40.523730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:40.524738] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:40.524772] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667965, retry interval=1000000, ret=-4179) [2022-08-08 10:55:40.552375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:40.593836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:40.613751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:40.624200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.624924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:40.665680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:40.678734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.694468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:40.706779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:40.726534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.751642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:40.765352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=214] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:40.855653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:40.919580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:41.092823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:41.092844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:41.107587] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:41.110832] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:41.110897] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:41.123884] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:41.298036] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11920, time_dist: do_cluster_heartbeat=11, generate_cluster_version=11904) [2022-08-08 10:55:41.345350] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] clock out of order(cur_ts=1659927341345316, cur_ts_=1659927341321635, delta=23681) [2022-08-08 10:55:41.535447] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:41.536770] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:41.536907] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667959, retry interval=1000000, ret=-4179) [2022-08-08 10:55:41.550300] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:41.552362] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:41.552421] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667964, retry interval=1000000, ret=-4179) [2022-08-08 10:55:41.686222] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] clock out of order(cur_ts=1659927341686215, cur_ts_=1659927341659573, delta=26642) [2022-08-08 10:55:42.056811] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927342056801, cur_ts_=1659927342018781, delta=38020) [2022-08-08 10:55:42.124454] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:42.129700] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:42.129899] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:42.134949] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18996) [2022-08-08 10:55:42.135826] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24951, time_dist: do_cluster_heartbeat=7, generate_cluster_version=24934, cluster_service_self_check=5) [2022-08-08 10:55:42.136506] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=187] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24772, time_dist: do_cluster_heartbeat=13, generate_cluster_version=24756) [2022-08-08 10:55:42.356461] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927342356454, cur_ts_=1659927342334805, delta=21649) [2022-08-08 10:55:42.540574] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC606A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:42.559400] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:42.559451] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667958, retry interval=1000000, ret=-4179) [2022-08-08 10:55:42.574646] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:42.576068] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:42.576116] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667963, retry interval=1000000, ret=-4179) [2022-08-08 10:55:42.783718] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927342783710, cur_ts_=1659927342751001, delta=32709) [2022-08-08 10:55:42.924163] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11911) [2022-08-08 10:55:42.927150] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15734, time_dist: generate_cluster_version=15729) [2022-08-08 10:55:42.935534] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23277, time_dist: generate_cluster_version=23272) [2022-08-08 10:55:43.022156] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] clock out of order(cur_ts=1659927343022146, cur_ts_=1659927342999429, delta=22717) [2022-08-08 10:55:43.142372] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:43.151374] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:43.151415] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:43.297116] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927343297106, cur_ts_=1659927343268610, delta=28496) [2022-08-08 10:55:43.423556] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659927343423546, cur_ts_=1659927343383881, delta=39665) [2022-08-08 10:55:43.559439] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=743] [dc=0] clock out of order(cur_ts=1659927343559427, cur_ts_=1659927343471533, delta=87894) [2022-08-08 10:55:43.671191] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] clock out of order(cur_ts=1659927343671181, cur_ts_=1659927343606580, delta=64601) [2022-08-08 10:55:43.720700] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=163996) [2022-08-08 10:55:43.727684] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=6884] [dc=0] slow flush(flush_time=64, io_time=170967, file_id=237, flush_start_offset=49602634, file_offset=49603255) [2022-08-08 10:55:43.734962] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:7298207458387846385, inc:84089181, addr:"192.168.11.128:2882", t:1659927343471533}, log_id=8300531, used_time=226615) [2022-08-08 10:55:43.752472] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=244766, time_dist: do_cluster_heartbeat=12, generate_cluster_version=244748) [2022-08-08 10:55:43.800859] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927343800846, cur_ts_=1659927343755773, delta=45073) [2022-08-08 10:55:43.815809] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:43.815852] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667962, retry interval=1000000, ret=-4179) [2022-08-08 10:55:43.850451] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=122711) [2022-08-08 10:55:43.862850] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=11798] [dc=0] slow flush(flush_time=553, io_time=134539, file_id=237, flush_start_offset=49603255, file_offset=49603871) [2022-08-08 10:55:43.862960] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=26] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:4372583535091634278, inc:84089180, addr:"192.168.11.128:2882", t:1659927343471533}, log_id=8315124, used_time=354585) [2022-08-08 10:55:43.891687] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28277] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:43.891742] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667957, retry interval=1000000, ret=-4179) [2022-08-08 10:55:43.888776] WARN [CLOG] submit_log_succ_cb (ob_log_task.cpp:323) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=25550] [dc=0] transaction on success costs(on_success_duration=25827, pkey={tid:1099511628002, partition_id:0, part_cnt:0}, log_id=8315124) [2022-08-08 10:55:43.888882] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=383216, time_dist: do_cluster_heartbeat=99, generate_cluster_version=383113) [2022-08-08 10:55:43.891817] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=2997] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511628002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8315124, proposal_id:{time_to_usec:1659815620290075, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:49603303, size:568}, after_consume_timestamp:1659927343856424}, end_time=1659927343891124, begin_time=1659927343856424, cost_time=34700) [2022-08-08 10:55:43.891278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:43.973715] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927343973700, cur_ts_=1659927343949743, delta=23957) [2022-08-08 10:55:43.973856] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=1023] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=109968) [2022-08-08 10:55:43.975930] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=2047] [dc=0] slow flush(flush_time=16, io_time=112022, file_id=237, flush_start_offset=49603871, file_offset=49605132) [2022-08-08 10:55:43.976027] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=6] [dc=0] transaction log sync use too much time(partition={tid:1099511627916, partition_id:0, part_cnt:0}, trans_id={hash:17469215590332141664, inc:84089192, addr:"192.168.11.128:2882", t:1659927343755000}, log_id=987603, used_time=157176) [2022-08-08 10:55:43.976102] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=5] [dc=0] [trans sampling] (TRACE=begin_ts=1659927343818299 2022-08-08 02:55:43.818299|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:17469215590332141664, inc:84089192, addr:{ip:"192.168.11.128", port:2882}, t:1659927343755000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=10 ret:0, left_time:10000000, uref:1073741825|[start_task] u=9 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7C0A-0-0, start_time:1659927343818017, snapshot:1659927343804033|[end_task] u=482 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927343818807|[calc_checksum_by_commit] u=22 ret:0, checksum:2509902709, checksum_log_ts:9223372036854775807|[submit_log] u=26 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=157182 trans_version:1659927343804034, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987603, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=157769) [2022-08-08 10:55:43.976143] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=5] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:463094049857943304, inc:84089195, addr:"192.168.11.128:2882", t:1659927343856424}, log_id=8300532, used_time=118951) [2022-08-08 10:55:43.977175] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=121246, time_dist: do_cluster_heartbeat=9, generate_cluster_version=121233) [2022-08-08 10:55:44.004698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.039421] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=63464) [2022-08-08 10:55:44.042186] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.042211] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=2910] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=94221, time_dist: do_cluster_heartbeat=13, generate_cluster_version=94206) [2022-08-08 10:55:44.052052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.067714] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28247) [2022-08-08 10:55:44.079891] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45345, time_dist: do_cluster_heartbeat=8, generate_cluster_version=45318) [2022-08-08 10:55:44.109260] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] clock out of order(cur_ts=1659927344109249, cur_ts_=1659927344087723, delta=21526) [2022-08-08 10:55:44.139638] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=25546) [2022-08-08 10:55:44.166569] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:44.201397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.204277] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=64593) [2022-08-08 10:55:44.204637] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=93361, time_dist: do_cluster_heartbeat=1465, generate_cluster_version=91893) [2022-08-08 10:55:44.206885] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:44.206914] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:44.212639] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=208] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=73822, time_dist: generate_cluster_version=73819) [2022-08-08 10:55:44.235464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:44.247140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.247446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.257873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:44.264471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.278669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:44.340576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.386697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:44.391092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.403994] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.406981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.411995] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.412140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:44.439023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.439082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:44.462594] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.481359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:44.481882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.485337] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.487339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.492270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.501276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.507284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:44.507385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.507418] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.513127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.517143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.517745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.553347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.557892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.558595] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10547, time_dist: generate_cluster_version=10544) [2022-08-08 10:55:44.571208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.595962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.618339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:44.630821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.641298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.642766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.645317] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:44.685206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.688774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:44.743263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:44.750886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:44.769088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.825179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.828140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.833377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:44.834786] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:44.834827] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667961, retry interval=1000000, ret=-4179) [2022-08-08 10:55:44.846032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:44.855323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:44.893906] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49E4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:44.918888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:44.921054] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:44.921097] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667956, retry interval=1000000, ret=-4179) [2022-08-08 10:55:44.949744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:45.019132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:45.139690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:45.210810] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:45.230165] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:45.230206] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:45.290922] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26501] [dc=0] clock out of order(cur_ts=1659927345290915, cur_ts_=1659927345250632, delta=40283) [2022-08-08 10:55:45.292419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:45.836625] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48A4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:45.858408] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:45.858447] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667960, retry interval=1000000, ret=-4179) [2022-08-08 10:55:45.923260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4867-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:45.923529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4867-0-0] [lt=131] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:45.923549] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4867-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:45.923883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4867-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:45.923950] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4867-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:45.942410] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:45.942564] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] task process failed, start retry(max retry time=9223372036854667955, retry interval=1000000, ret=-4179) [2022-08-08 10:55:46.176394] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:46.199742] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:46.199838] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:46.200033] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:46.200110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:46.233365] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:46.236868] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:46.236907] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:46.555122] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10701, time_dist: generate_cluster_version=10698) [2022-08-08 10:55:46.710492] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927346710487, cur_ts_=1659927346679893, delta=30594) [2022-08-08 10:55:46.771963] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927346766792 2022-08-08 02:55:46.766792|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:2589538646739041936, inc:84089309, addr:{ip:"192.168.11.128", port:2882}, t:1659927346766000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=10 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABC73-0-0, start_time:1659927346766544, snapshot:1659927346754712|[end_task] u=1715 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=14 stc:1659927346768533|[calc_checksum_by_commit] u=22 ret:0, checksum:2120263050, checksum_log_ts:9223372036854775807|[submit_log] u=57 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2840 trans_version:1659927346766069, uref:1073741825|[exiting] u=115 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=75 ret:0, log_type:64, log_id:8315140, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=5 arg1:140085233547952|total_timeu=4867) [2022-08-08 10:55:46.860479] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49E7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:46.877581] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:46.879078] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:46.879130] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667959, retry interval=1000000, ret=-4179) [2022-08-08 10:55:46.964720] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:46.966174] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:46.966226] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667954, retry interval=1000000, ret=-4179) [2022-08-08 10:55:47.239144] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:47.242460] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:47.242497] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:47.884002] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C486A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:47.899105] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:47.900481] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:47.900518] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667958, retry interval=1000000, ret=-4179) [2022-08-08 10:55:47.968663] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48A7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:47.996251] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:47.997055] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=680] [dc=0] task process failed, start retry(max retry time=9223372036854667953, retry interval=1000000, ret=-4179) [2022-08-08 10:55:48.247177] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:48.250481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:48.250525] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:48.857474] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927348857465, cur_ts_=1659927348808670, delta=48795) [2022-08-08 10:55:48.873822] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=82938) [2022-08-08 10:55:48.903159] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=29307] [dc=0] slow flush(flush_time=18, io_time=112259, file_id=237, flush_start_offset=49654507, file_offset=49655123) [2022-08-08 10:55:48.904716] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:6366662126915508939, inc:84089387, addr:"192.168.11.128:2882", t:1659927348778662}, log_id=8315150, used_time=114784) [2022-08-08 10:55:48.905288] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=116943, time_dist: generate_cluster_version=116940) [2022-08-08 10:55:48.937534] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34315) [2022-08-08 10:55:48.937747] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:8762940876919818384, inc:84089388, addr:"192.168.11.128:2882", t:1659927348778662}, log_id=8300558, used_time=147195) [2022-08-08 10:55:48.937948] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=148057, time_dist: generate_cluster_version=148055) [2022-08-08 10:55:48.954139] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:48.954206] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667957, retry interval=1000000, ret=-4179) [2022-08-08 10:55:49.016572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:49.018072] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:49.018107] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667952, retry interval=1000000, ret=-4179) [2022-08-08 10:55:49.253449] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:49.258706] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:49.258748] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:49.417454] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=13130) [2022-08-08 10:55:49.418386] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14821, time_dist: generate_cluster_version=14818) [2022-08-08 10:55:49.630200] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10667, time_dist: generate_cluster_version=10662) [2022-08-08 10:55:49.784773] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927349784765, cur_ts_=1659927349746184, delta=38581) [2022-08-08 10:55:49.830236] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:55:49.830267] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:55:49.830754] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=4] [dc=0] [trans sampling] (TRACE=begin_ts=1659927349826990 2022-08-08 02:55:49.826990|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:7151350320054593388, inc:84089423, addr:{ip:"192.168.11.128", port:2882}, t:1659927349825000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABC82-0-0, start_time:1659927349826878, snapshot:1659927349772870|[end_task] u=161 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927349827169|[calc_checksum_by_commit] u=20 ret:0, checksum:2538530075, checksum_log_ts:9223372036854775807|[submit_log] u=35 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=15 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3429 trans_version:1659927349825399, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8315155, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=3707) [2022-08-08 10:55:49.971396] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:49.972469] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:49.972503] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667956, retry interval=1000000, ret=-4179) [2022-08-08 10:55:50.057357] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927350057349, cur_ts_=1659927350017234, delta=40115) [2022-08-08 10:55:50.066624] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:50.068048] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:50.068099] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667951, retry interval=1000000, ret=-4179) [2022-08-08 10:55:50.216683] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14741568}) [2022-08-08 10:55:50.239273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.239831] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] clock out of order(cur_ts=1659927350239825, cur_ts_=1659927350216182, delta=23643) [2022-08-08 10:55:50.249488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.258851] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15611) [2022-08-08 10:55:50.260505] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:50.264087] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:50.264130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:50.278723] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13795) [2022-08-08 10:55:50.343845] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=51517) [2022-08-08 10:55:50.345631] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=53918, time_dist: generate_cluster_version=53914) [2022-08-08 10:55:50.351746] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31838, time_dist: do_cluster_heartbeat=12, generate_cluster_version=31821) [2022-08-08 10:55:50.407681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.420875] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927348268516, end_time=1659927350420863, elapsed_time=2152347, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:55:50.477325] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] clock out of order(cur_ts=1659927350477320, cur_ts_=1659927350433407, delta=43913) [2022-08-08 10:55:50.544093] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=68087) [2022-08-08 10:55:50.544598] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=72238, time_dist: print_stat=6, do_cluster_heartbeat=3, generate_cluster_version=72224, cluster_service_self_check=3) [2022-08-08 10:55:50.551137] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=74705, time_dist: print_stat=6, do_cluster_heartbeat=2, generate_cluster_version=74678, cluster_service_self_check=12) [2022-08-08 10:55:50.577509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:50.600793] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3960] [dc=0] clock out of order(cur_ts=1659927350600785, cur_ts_=1659927350550288, delta=50497) [2022-08-08 10:55:50.613683] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12789) [2022-08-08 10:55:50.615532] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16970, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16956) [2022-08-08 10:55:50.647409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:50.647970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=166] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:50.649183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.650364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:50.684568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:50.685069] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=70433) [2022-08-08 10:55:50.716905] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=105442, time_dist: do_cluster_heartbeat=11, generate_cluster_version=105426) [2022-08-08 10:55:50.718465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:50.718470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=418] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.721428] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927350721421, cur_ts_=1659927350684804, delta=36617) [2022-08-08 10:55:50.727684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:50.750371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.772057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:50.781656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:50.875549] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:50.938713] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:50.993027] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927350993018, cur_ts_=1659927350972089, delta=20929) [2022-08-08 10:55:51.006204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:51.008616] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:51.008656] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667955, retry interval=1000000, ret=-4179) [2022-08-08 10:55:51.114545] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927351114539, cur_ts_=1659927351092669, delta=21870) [2022-08-08 10:55:51.114742] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=41958) [2022-08-08 10:55:51.115031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:51.115956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:51.118945] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=46879, time_dist: generate_cluster_version=46876) [2022-08-08 10:55:51.122415] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:51.122471] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667950, retry interval=1000000, ret=-4179) [2022-08-08 10:55:51.172023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:51.266713] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:51.271841] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:51.272013] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:52.050124] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927352050112, cur_ts_=1659927352026801, delta=23311) [2022-08-08 10:55:52.073391] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:52.074947] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:52.074997] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667954, retry interval=1000000, ret=-4179) [2022-08-08 10:55:52.075048] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15069) [2022-08-08 10:55:52.075673] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20453, time_dist: do_cluster_heartbeat=18, generate_cluster_version=20433) [2022-08-08 10:55:52.147509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:52.147554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:52.148922] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:52.148971] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667949, retry interval=1000000, ret=-4179) [2022-08-08 10:55:52.250341] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E2E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.250400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E2E-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.250611] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E2E-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.251006] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E2E-0-0] [lt=360] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.251271] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.251314] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.251345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.251356] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:55:52.275567] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:52.278771] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:52.278793] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:52.485548] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=150] [dc=0] clock out of order(cur_ts=1659927352485541, cur_ts_=1659927352446226, delta=39315) [2022-08-08 10:55:52.650363] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:52.722172] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927352722153, cur_ts_=1659927352692065, delta=30088) [2022-08-08 10:55:52.762861] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=270] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39973) [2022-08-08 10:55:52.763827] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=59289, time_dist: do_cluster_heartbeat=209, generate_cluster_version=59077) [2022-08-08 10:55:52.767151] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=56565, time_dist: do_cluster_heartbeat=8, generate_cluster_version=56552) [2022-08-08 10:55:52.828241] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=20] [dc=0] [trans sampling] (TRACE=begin_ts=1659927352826343 2022-08-08 02:55:52.826343|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:9705265134634139399, inc:84089539, addr:{ip:"192.168.11.128", port:2882}, t:1659927352826000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABC91-0-0, start_time:1659927352826219, snapshot:1659927352822529|[end_task] u=144 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927352826505|[calc_checksum_by_commit] u=19 ret:0, checksum:3503182525, checksum_log_ts:9223372036854775807|[submit_log] u=677 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=956 trans_version:1659927352826212, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315170, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=1855) [2022-08-08 10:55:53.116066] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659927353116056, cur_ts_=1659927353083283, delta=32773) [2022-08-08 10:55:53.127106] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10812, time_dist: do_cluster_heartbeat=15, generate_cluster_version=10792) [2022-08-08 10:55:53.139162] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22026, time_dist: do_cluster_heartbeat=6, generate_cluster_version=22017) [2022-08-08 10:55:53.231948] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:53.231989] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667953, retry interval=1000000, ret=-4179) [2022-08-08 10:55:53.260864] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:53.260883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:53.262302] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:53.262350] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667948, retry interval=1000000, ret=-4179) [2022-08-08 10:55:53.281458] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:53.284850] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:53.284888] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:53.914878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=585] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:54.022767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.058795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.077092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.219993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.250975] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:54.251250] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:54.252885] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:54.252975] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] task process failed, start retry(max retry time=9223372036854667952, retry interval=1000000, ret=-4179) [2022-08-08 10:55:54.256739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:54.264258] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48B0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:54.267327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.275026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=276] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.277731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:54.281400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:54.281545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:54.282706] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:54.282765] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667947, retry interval=1000000, ret=-4179) [2022-08-08 10:55:54.287843] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:54.287945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=2426] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.293411] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:54.293436] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:54.298512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:54.360733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.404275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:54.414629] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.425108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.426802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.434369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.437251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:54.456160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:54.463444] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.487365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.496705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:55:54.503596] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.508269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.510706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.510798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.521370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.529127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:55:54.533972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.534971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.539225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.539650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.541943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.576759] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=351] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.577105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.586996] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.611522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.638779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:54.652239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.662524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.662926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.672966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:55:54.712077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:54.712158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.761542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:55:54.772863] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:54.794502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.850131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.850836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.867291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:54.877179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:54.944245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:55:54.971955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:55.035826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:55:55.158504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:55.258118] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49F5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:55.258274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49F5-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:55.276203] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:55.276253] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667951, retry interval=1000000, ret=-4179) [2022-08-08 10:55:55.295854] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:55.299536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:55.299560] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:55.309480] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:55.309512] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667946, retry interval=1000000, ret=-4179) [2022-08-08 10:55:55.309736] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:55:55.311827] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1158] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17555, time_dist: generate_cluster_version=17552) [2022-08-08 10:55:55.564628] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:55:55.857501] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927355857491, cur_ts_=1659927355833109, delta=24382) [2022-08-08 10:55:55.859154] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] [trans sampling] (TRACE=begin_ts=1659927355858381 2022-08-08 02:55:55.858381|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:5331317508580952144, inc:84089642, addr:{ip:"192.168.11.128", port:2882}, t:1659927355833000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5ED4-0-0, start_time:1659927355858268, snapshot:1659927355857313|[end_task] u=120 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927355858518|[calc_checksum_by_commit] u=19 ret:0, checksum:6321634, checksum_log_ts:9223372036854775807|[submit_log] u=58 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=474 trans_version:1659927355858538, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8300594, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=724) [2022-08-08 10:55:56.236105] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:56.278051] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48B3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:56.278188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48B3-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:56.296973] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:56.297033] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667950, retry interval=1000000, ret=-4179) [2022-08-08 10:55:56.311526] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:56.315515] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:56.315535] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:56.336353] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:56.336381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:56.338495] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:56.338535] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667945, retry interval=1000000, ret=-4179) [2022-08-08 10:55:56.692879] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12066) [2022-08-08 10:55:56.693157] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13043, time_dist: generate_cluster_version=13041) [2022-08-08 10:55:57.298490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC607F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.298734] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC607F-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.314872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.315020] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.316598] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:57.316636] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667949, retry interval=1000000, ret=-4179) [2022-08-08 10:55:57.318466] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:57.323037] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:57.323075] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:57.341002] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49F8-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.341367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49F8-0-0] [lt=135] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.358977] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.359099] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:57.360334] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:57.360383] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667944, retry interval=1000000, ret=-4179) [2022-08-08 10:55:57.383408] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:55:58.318341] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4879-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:58.318491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4879-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:58.335334] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:58.355594] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927358355551, cur_ts_=1659927358318258, delta=37293) [2022-08-08 10:55:58.360946] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:58.360971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:58.391571] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:58.391626] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667943, retry interval=1000000, ret=-4179) [2022-08-08 10:55:58.401864] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:58.401899] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667948, retry interval=1000000, ret=-4179) [2022-08-08 10:55:58.491890] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14976, time_dist: generate_cluster_version=14972) [2022-08-08 10:55:58.668732] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10940) [2022-08-08 10:55:58.668944] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12043, time_dist: generate_cluster_version=12040) [2022-08-08 10:55:58.713818] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25497) [2022-08-08 10:55:58.714144] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26581, time_dist: generate_cluster_version=26579) [2022-08-08 10:55:58.883074] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927358883064, cur_ts_=1659927358856272, delta=26792) [2022-08-08 10:55:58.897631] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927358896853 2022-08-08 02:55:58.896853|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:18440960036424168676, inc:84089767, addr:{ip:"192.168.11.128", port:2882}, t:1659927358896000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5EE3-0-0, start_time:1659927358896661, snapshot:1659927358882882|[end_task] u=141 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927358897013|[calc_checksum_by_commit] u=23 ret:0, checksum:3065556646, checksum_log_ts:9223372036854775807|[submit_log] u=46 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=464 trans_version:1659927358897037, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8300609, logging:0, pending:0, uref:1|[end_trans_cb] u=24 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=739) [2022-08-08 10:55:59.067073] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11249) [2022-08-08 10:55:59.067344] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12227, time_dist: do_cluster_heartbeat=11, generate_cluster_version=12213) [2022-08-08 10:55:59.253089] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927359253083, cur_ts_=1659927359227652, delta=25431) [2022-08-08 10:55:59.372562] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:55:59.376249] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:55:59.376295] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:55:59.409260] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:59.409298] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667942, retry interval=1000000, ret=-4179) [2022-08-08 10:55:59.434279] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:59.434389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:55:59.436032] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:55:59.436069] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667947, retry interval=1000000, ret=-4179) [2022-08-08 10:55:59.715922] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:55:59.715945] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:55:59.955057] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14745824}) [2022-08-08 10:56:00.084927] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17689) [2022-08-08 10:56:00.085426] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19380, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19369) [2022-08-08 10:56:00.108158] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10185) [2022-08-08 10:56:00.115391] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18163, time_dist: generate_cluster_version=18159) [2022-08-08 10:56:00.176029] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=466] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927358276404, end_time=1659927360175957, elapsed_time=1899553, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:56:00.269794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.273749] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.377979] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:00.384202] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:00.384249] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:00.411705] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48B9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:00.431980] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:00.432032] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667941, retry interval=1000000, ret=-4179) [2022-08-08 10:56:00.432675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.456409] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:00.456544] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] task process failed, start retry(max retry time=9223372036854667946, retry interval=1000000, ret=-4179) [2022-08-08 10:56:00.597735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:00.665692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:00.665848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=175] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:00.670556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.670606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:00.708946] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:00.743491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.743498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:00.743682] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:00.774733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.795613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=857] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:00.807460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:00.899806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:00.960340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:01.138940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:01.139149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:01.190864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:01.386759] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:01.390387] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:01.390423] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:01.434106] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=915] [dc=0] clock out of order(cur_ts=1659927361434102, cur_ts_=1659927361408993, delta=25109) [2022-08-08 10:56:01.435686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.435868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.436003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.436103] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.436370] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.436584] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.436640] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.436778] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C49FE-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.467439] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:01.467495] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667940, retry interval=1000000, ret=-4179) [2022-08-08 10:56:01.485392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.485476] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:01.488128] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:01.488180] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667945, retry interval=1000000, ret=-4179) [2022-08-08 10:56:01.488237] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18924) [2022-08-08 10:56:01.940273] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927361939481 2022-08-08 02:56:01.939481|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:5054711928562609720, inc:84089876, addr:{ip:"192.168.11.128", port:2882}, t:1659927361938000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5EF2-0-0, start_time:1659927361939358, snapshot:1659927361934433|[end_task] u=123 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927361939622|[calc_checksum_by_commit] u=20 ret:0, checksum:230214271, checksum_log_ts:9223372036854775807|[submit_log] u=67 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=484 trans_version:1659927361939643, uref:1073741825|[exiting] u=9 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8300624, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=745) [2022-08-08 10:56:02.393238] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:02.398260] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:02.398299] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:02.488690] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:02.488805] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667939, retry interval=1000000, ret=-4179) [2022-08-08 10:56:02.511850] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:02.511886] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667944, retry interval=1000000, ret=-4179) [2022-08-08 10:56:02.524401] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14070, time_dist: do_cluster_heartbeat=8, generate_cluster_version=14057) [2022-08-08 10:56:03.295287] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E4F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:03.295442] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:03.295491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:03.295847] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:03.295874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:03.401094] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:03.404215] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:03.404252] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:03.490900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A01-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:03.491087] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A01-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:03.508844] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:03.508882] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667938, retry interval=1000000, ret=-4179) [2022-08-08 10:56:03.530623] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:03.530679] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667943, retry interval=1000000, ret=-4179) [2022-08-08 10:56:03.933546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:04.039735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.080781] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.101510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.235313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.281479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:04.283284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.293469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.293534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:04.311908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.314481] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:04.377699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.406472] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:04.409640] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:04.409696] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:04.427673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:04.439788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.447721] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.450174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.452128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.460551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:04.480461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:04.485350] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.503216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.511110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4882-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:04.511349] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4882-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:04.513842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:04.522041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.524112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.534115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.534555] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:04.534714] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667937, retry interval=1000000, ret=-4179) [2022-08-08 10:56:04.542051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.542882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.549092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.549472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:04.555167] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:04.555275] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667942, retry interval=1000000, ret=-4179) [2022-08-08 10:56:04.555407] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19959) [2022-08-08 10:56:04.556899] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20512, time_dist: generate_cluster_version=20508) [2022-08-08 10:56:04.558698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.559286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.563188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.564905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.595486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.596707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.606946] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.626719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.660124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:04.668446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.680913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=239] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.684221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=622] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.690334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:04.732926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=673] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.733010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=577] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:04.785674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=603] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:04.796932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:04.813129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.868325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.871677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.885619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:04.902452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:04.964490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:04.991260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:05.057263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:05.179262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:05.248261] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927365248252, cur_ts_=1659927365222890, delta=25362) [2022-08-08 10:56:05.331900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:05.411844] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:05.415961] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:05.416017] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:05.558177] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:05.558213] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667936, retry interval=1000000, ret=-4179) [2022-08-08 10:56:05.575010] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:05.575132] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667941, retry interval=1000000, ret=-4179) [2022-08-08 10:56:06.298355] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:06.418304] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:06.421741] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:06.421775] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:06.589249] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:06.589391] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667935, retry interval=1000000, ret=-4179) [2022-08-08 10:56:06.618528] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:06.618682] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667940, retry interval=1000000, ret=-4179) [2022-08-08 10:56:07.424431] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:07.429092] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:07.429120] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:07.610284] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:07.610320] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667934, retry interval=1000000, ret=-4179) [2022-08-08 10:56:07.638316] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:07.638496] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667939, retry interval=1000000, ret=-4179) [2022-08-08 10:56:08.312981] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15992) [2022-08-08 10:56:08.313195] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16979, time_dist: do_cluster_heartbeat=7, generate_cluster_version=16969) [2022-08-08 10:56:08.399800] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10618) [2022-08-08 10:56:08.399993] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11616, time_dist: generate_cluster_version=11613) [2022-08-08 10:56:08.432050] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:08.436122] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:08.436159] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:08.468095] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11903) [2022-08-08 10:56:08.488083] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18299) [2022-08-08 10:56:08.519776] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12520) [2022-08-08 10:56:08.520073] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14609, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14598) [2022-08-08 10:56:08.551131] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18387) [2022-08-08 10:56:08.551347] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19559, time_dist: generate_cluster_version=19557) [2022-08-08 10:56:08.635881] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=193] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:08.636259] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667933, retry interval=1000000, ret=-4179) [2022-08-08 10:56:08.676375] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:08.676943] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] task process failed, start retry(max retry time=9223372036854667938, retry interval=1000000, ret=-4179) [2022-08-08 10:56:08.790096] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46235) [2022-08-08 10:56:08.790730] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=48197, time_dist: generate_cluster_version=48193) [2022-08-08 10:56:08.825728] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35585) [2022-08-08 10:56:08.827346] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=42006, time_dist: do_cluster_heartbeat=8, generate_cluster_version=41996) [2022-08-08 10:56:08.828865] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] clock out of order(cur_ts=1659927368828860, cur_ts_=1659927368800995, delta=27865) [2022-08-08 10:56:09.018557] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927369018551, cur_ts_=1659927368985673, delta=32878) [2022-08-08 10:56:09.019007] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33350) [2022-08-08 10:56:09.019194] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34313, time_dist: generate_cluster_version=34312) [2022-08-08 10:56:09.065686] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=22] [dc=0] flush_cb cost too much time(partition_key_={tid:1100611139453778, partition_id:0, part_cnt:0}, arg={log_type:202, log_id:4, proposal_id:{time_to_usec:1659926840155590, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:49844538, size:494}, after_consume_timestamp:1659927369045733}, end_time=1659927369062694, begin_time=1659927369045733, cost_time=16961) [2022-08-08 10:56:09.135017] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11342) [2022-08-08 10:56:09.142958] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20029, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20018) [2022-08-08 10:56:09.153712] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18655) [2022-08-08 10:56:09.153911] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28426, time_dist: do_cluster_heartbeat=10, generate_cluster_version=28414) [2022-08-08 10:56:09.438414] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:09.442471] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:09.442511] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:09.657667] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:09.657720] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667932, retry interval=1000000, ret=-4179) [2022-08-08 10:56:09.699118] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:09.699171] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667937, retry interval=1000000, ret=-4179) [2022-08-08 10:56:09.716183] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:56:09.716202] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:56:09.969136] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14750208}) [2022-08-08 10:56:10.161521] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927368270260, end_time=1659927370161508, elapsed_time=1891248, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:56:10.292784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.294001] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.343384] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=255] [dc=0] clock out of order(cur_ts=1659927370343379, cur_ts_=1659927370316303, delta=27076) [2022-08-08 10:56:10.453933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.455302] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927370455299, cur_ts_=1659927370434092, delta=21207) [2022-08-08 10:56:10.455944] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:10.469492] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:10.469567] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:10.613853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:10.681035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:10.683762] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:10.683826] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667931, retry interval=1000000, ret=-4179) [2022-08-08 10:56:10.688054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:10.690412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:10.690466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.726687] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:10.741107] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:10.741154] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667936, retry interval=1000000, ret=-4179) [2022-08-08 10:56:10.760569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.767383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:10.767474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:10.798533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.820537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:10.833993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:10.918077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:10.981084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:11.119136] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927371119128, cur_ts_=1659927371097109, delta=22019) [2022-08-08 10:56:11.156143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:11.160757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:11.209112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:11.472131] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:11.476220] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:11.476245] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:11.707270] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:11.707337] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667930, retry interval=1000000, ret=-4179) [2022-08-08 10:56:11.775592] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13106] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:11.775653] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667935, retry interval=1000000, ret=-4179) [2022-08-08 10:56:12.478700] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:12.481783] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:12.481828] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:12.650574] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:12.727707] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:12.727745] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667929, retry interval=1000000, ret=-4179) [2022-08-08 10:56:12.802707] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=916] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:12.802808] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667934, retry interval=1000000, ret=-4179) [2022-08-08 10:56:13.484382] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:13.487560] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:13.487604] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:13.747172] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:13.747237] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667928, retry interval=1000000, ret=-4179) [2022-08-08 10:56:13.823704] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=3089] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:13.823775] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667933, retry interval=1000000, ret=-4179) [2022-08-08 10:56:13.949763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:14.055960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.103377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.118419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.253235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.300460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:14.305906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.316440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.316926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:14.327314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.337322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:14.399249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.443376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:14.459368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.472058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.472147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.472204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.482562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:14.490339] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:14.495120] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:14.495166] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:14.499166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:14.510202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.519178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.529801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:14.541445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.541512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.559305] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.560924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.566208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.571444] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.571488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:14.576149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.584351] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.587582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.590100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.614804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.620289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.623388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.645134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.676532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:14.684973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.698148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.700217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.706749] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:14.748850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.750835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:14.765804] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:14.765840] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667927, retry interval=1000000, ret=-4179) [2022-08-08 10:56:14.805925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:14.815310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:14.832100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.856959] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:14.857018] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667932, retry interval=1000000, ret=-4179) [2022-08-08 10:56:14.893203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.893250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.903719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:14.921478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:14.985888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:15.016758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:15.079085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:15.198760] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:15.357106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=161] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:15.499225] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:15.502442] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:15.502479] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:15.591259] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927375591247, cur_ts_=1659927375559253, delta=31994) [2022-08-08 10:56:15.614055] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:56:15.779086] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927375779080, cur_ts_=1659927375749665, delta=29415) [2022-08-08 10:56:15.795094] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:15.795155] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667926, retry interval=1000000, ret=-4179) [2022-08-08 10:56:15.875967] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:15.876730] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=283] [dc=0] task process failed, start retry(max retry time=9223372036854667931, retry interval=1000000, ret=-4179) [2022-08-08 10:56:16.359992] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:16.509293] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:16.512876] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:16.512949] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:16.814804] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:16.814865] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667925, retry interval=1000000, ret=-4179) [2022-08-08 10:56:16.862535] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927376862530, cur_ts_=1659927376834152, delta=28378) [2022-08-08 10:56:16.905654] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:16.905718] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667930, retry interval=1000000, ret=-4179) [2022-08-08 10:56:17.515493] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:17.518803] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:17.518879] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:17.836349] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:17.836525] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] task process failed, start retry(max retry time=9223372036854667924, retry interval=1000000, ret=-4179) [2022-08-08 10:56:17.931088] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:17.931122] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854667929, retry interval=1000000, ret=-4179) [2022-08-08 10:56:18.536217] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:18.540237] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927378540228, cur_ts_=1659927378517339, delta=22889) [2022-08-08 10:56:18.544253] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:18.546043] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=1770] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:18.856900] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:18.856955] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667923, retry interval=1000000, ret=-4179) [2022-08-08 10:56:18.951502] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:18.951535] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667928, retry interval=1000000, ret=-4179) [2022-08-08 10:56:19.389385] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24837) [2022-08-08 10:56:19.389739] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=36024, time_dist: do_cluster_heartbeat=8, generate_cluster_version=36010, cluster_service_self_check=3) [2022-08-08 10:56:19.390840] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22438, time_dist: generate_cluster_version=22434) [2022-08-08 10:56:19.554188] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:19.559745] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:19.559801] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:19.567340] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13710, time_dist: do_cluster_heartbeat=6, generate_cluster_version=13701) [2022-08-08 10:56:19.719528] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:56:19.719548] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:56:19.860981] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14754592}) [2022-08-08 10:56:19.879205] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:19.879311] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667922, retry interval=1000000, ret=-4179) [2022-08-08 10:56:19.971804] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:19.971857] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667927, retry interval=1000000, ret=-4179) [2022-08-08 10:56:20.024366] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18892) [2022-08-08 10:56:20.027514] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23080, time_dist: generate_cluster_version=23075) [2022-08-08 10:56:20.027697] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23460, time_dist: generate_cluster_version=23458) [2022-08-08 10:56:20.055189] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927378270395, end_time=1659927380055174, elapsed_time=1784779, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:56:20.309002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:20.315911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:20.401763] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927380401751, cur_ts_=1659927380329147, delta=72604) [2022-08-08 10:56:20.501622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:20.504718] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927380504713, cur_ts_=1659927380419479, delta=85234) [2022-08-08 10:56:20.544800] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20917) [2022-08-08 10:56:20.549761] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43011, time_dist: do_cluster_heartbeat=6, generate_cluster_version=42999) [2022-08-08 10:56:20.585805] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:20.586598] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=28590) [2022-08-08 10:56:20.593256] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35985, time_dist: do_cluster_heartbeat=11, generate_cluster_version=35970) [2022-08-08 10:56:20.595007] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:20.595053] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:20.603266] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10356) [2022-08-08 10:56:20.638148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:20.704765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:20.705800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:20.705832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:20.715245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:20.746447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:20.783692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:20.788735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:20.790439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:20.810516] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10484, time_dist: generate_cluster_version=10480) [2022-08-08 10:56:20.820637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:20.843188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=690] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:20.854291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:20.907512] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:20.907713] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] task process failed, start retry(max retry time=9223372036854667921, retry interval=1000000, ret=-4179) [2022-08-08 10:56:20.946199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:21.010578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:21.011545] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927381011540, cur_ts_=1659927380987760, delta=23780) [2022-08-08 10:56:21.011619] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20532) [2022-08-08 10:56:21.012563] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24621, time_dist: generate_cluster_version=24619) [2022-08-08 10:56:21.016644] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:21.016770] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667926, retry interval=1000000, ret=-4179) [2022-08-08 10:56:21.178717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:21.180118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:21.229062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=251] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:21.607511] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:21.610966] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:21.610991] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:21.932101] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:21.932163] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667920, retry interval=1000000, ret=-4179) [2022-08-08 10:56:22.036064] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:22.036115] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667925, retry interval=1000000, ret=-4179) [2022-08-08 10:56:22.651765] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927382651757, cur_ts_=1659927382603408, delta=48349) [2022-08-08 10:56:22.656278] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:22.762396] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:22.762424] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:22.769829] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=115096) [2022-08-08 10:56:22.775286] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=5430, io_time=115121, file_id=237, flush_start_offset=49966122, file_offset=49966744) [2022-08-08 10:56:22.776128] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:15405074797656740895, inc:84090661, addr:"192.168.11.128:2882", t:1659927382603408}, log_id=8300727, used_time=164533) [2022-08-08 10:56:22.780396] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=169498, time_dist: generate_cluster_version=169493) [2022-08-08 10:56:22.815920] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=40598) [2022-08-08 10:56:22.816429] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=12] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:4148494122431327589, inc:84090662, addr:"192.168.11.128:2882", t:1659927382603408}, log_id=1655596, used_time=194657) [2022-08-08 10:56:22.817055] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=32] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:4724394166967707972, inc:84090663, addr:"192.168.11.128:2882", t:1659927382603408}, log_id=8315318, used_time=159258) [2022-08-08 10:56:22.820762] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=163654, time_dist: do_cluster_heartbeat=6, generate_cluster_version=163644) [2022-08-08 10:56:22.860127] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17977) [2022-08-08 10:56:22.860376] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20685, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20675) [2022-08-08 10:56:22.947849] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927382947842, cur_ts_=1659927382912260, delta=35582) [2022-08-08 10:56:22.956436] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=80137) [2022-08-08 10:56:22.959044] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=83307, time_dist: do_cluster_heartbeat=11, generate_cluster_version=83292) [2022-08-08 10:56:22.973232] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:22.973419] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] task process failed, start retry(max retry time=9223372036854667919, retry interval=1000000, ret=-4179) [2022-08-08 10:56:23.034166] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=884] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22590) [2022-08-08 10:56:23.034355] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23340, time_dist: do_cluster_heartbeat=6, generate_cluster_version=23332) [2022-08-08 10:56:23.038429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48DC-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:23.038487] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48DC-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:23.060675] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:23.060865] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667924, retry interval=1000000, ret=-4179) [2022-08-08 10:56:23.467567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E8B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:23.468402] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:23.468620] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=110] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:23.592254] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927383592244, cur_ts_=1659927383570052, delta=22192) [2022-08-08 10:56:23.778007] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:23.781060] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:23.781081] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:23.970725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:23.993502] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:23.993571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667918, retry interval=1000000, ret=-4179) [2022-08-08 10:56:23.993661] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12690) [2022-08-08 10:56:23.994299] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14068, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14051, cluster_service_self_check=5) [2022-08-08 10:56:24.076956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.091973] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:24.092015] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667923, retry interval=1000000, ret=-4179) [2022-08-08 10:56:24.122794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.140089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.277740] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.321146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:24.321235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.341168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:24.342382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.352440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.352761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:24.417968] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.461111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:24.474681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.485897] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E8E-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.485940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E8E-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.486183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E8E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.486373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E8E-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.486540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.487073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.487242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:24.492113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.493058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.495828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.503907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:24.516786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:24.532834] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.534710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.547433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:24.559621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.561734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.579535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.583580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.585703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.588564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.590796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:24.596761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.600911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=636] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.607091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.612665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.635419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.638345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.644889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.664924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.696990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:24.704783] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.713302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.720812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.725443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:24.772819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.772835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:24.783498] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:24.786550] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:24.786601] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:24.824870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:24.835407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:24.848603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=6992] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.915963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.916217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.926834] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:24.953995] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:24.997491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48DF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:24.997527] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48DF-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:24.997541] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C48DF-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:25.005211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:25.018356] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:25.018416] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667917, retry interval=1000000, ret=-4179) [2022-08-08 10:56:25.039374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:25.104755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:25.115461] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:25.115912] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] task process failed, start retry(max retry time=9223372036854667922, retry interval=1000000, ret=-4179) [2022-08-08 10:56:25.214273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:25.382649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=286] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:25.494979] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E91-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495019] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E91-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495186] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E91-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495295] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E91-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495412] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.495965] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:25.790192] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:25.793288] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:25.793325] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:26.061569] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927386061560, cur_ts_=1659927386041461, delta=20099) [2022-08-08 10:56:26.078913] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:26.078949] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667916, retry interval=1000000, ret=-4179) [2022-08-08 10:56:26.180708] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:26.180768] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667921, retry interval=1000000, ret=-4179) [2022-08-08 10:56:26.182736] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=18] [dc=0] [trans sampling] (TRACE=begin_ts=1659927386181237 2022-08-08 02:56:26.181237|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:8217437480400202557, inc:84091573, addr:{ip:"192.168.11.128", port:2882}, t:1659927386180000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7C5D-0-0, start_time:1659927386181111, snapshot:1659927386179769|[end_task] u=104 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927386181359|[calc_checksum_by_commit] u=17 ret:0, checksum:1116186003, checksum_log_ts:9223372036854775807|[submit_log] u=55 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1220 trans_version:1659927386180717, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987686, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=1456) [2022-08-08 10:56:26.419319] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:26.504429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:26.507567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2913] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:26.507629] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:26.507645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:26.780268] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927386780259, cur_ts_=1659927386749896, delta=30363) [2022-08-08 10:56:26.795790] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:26.799392] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:26.799418] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:27.135290] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:27.135334] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667915, retry interval=1000000, ret=-4179) [2022-08-08 10:56:27.196746] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13985, time_dist: generate_cluster_version=13981) [2022-08-08 10:56:27.215158] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:27.215244] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] task process failed, start retry(max retry time=9223372036854667920, retry interval=1000000, ret=-4179) [2022-08-08 10:56:27.516192] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E97-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:27.516212] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E97-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:27.516218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E97-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:27.516222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E97-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:27.516238] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2E97-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:27.802061] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:27.805951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:27.805974] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:28.140522] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:28.140572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:28.160651] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:28.160731] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667914, retry interval=1000000, ret=-4179) [2022-08-08 10:56:28.239717] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:28.239767] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667919, retry interval=1000000, ret=-4179) [2022-08-08 10:56:28.808635] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:28.811862] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:28.811887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:29.168127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:29.168234] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:29.184956] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:29.185016] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667913, retry interval=1000000, ret=-4179) [2022-08-08 10:56:29.193706] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=31] [dc=0] [trans sampling] (TRACE=begin_ts=1659927389185876 2022-08-08 02:56:29.185876|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:10681329598991672937, inc:84091692, addr:{ip:"192.168.11.128", port:2882}, t:1659927389183000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7C62-0-0, start_time:1659927389185761, snapshot:1659927389171162|[end_task] u=101 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927389185994|[calc_checksum_by_commit] u=18 ret:0, checksum:80379804, checksum_log_ts:9223372036854775807|[submit_log] u=132 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=7478 trans_version:1659927389183739, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987691, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=7788) [2022-08-08 10:56:29.263567] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:29.263636] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667918, retry interval=1000000, ret=-4179) [2022-08-08 10:56:29.451222] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19756) [2022-08-08 10:56:29.451573] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21697, time_dist: generate_cluster_version=21694) [2022-08-08 10:56:29.452543] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12082, time_dist: generate_cluster_version=12079) [2022-08-08 10:56:29.556895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:29.557034] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:29.557239] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:29.557384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:29.720838] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:56:29.720933] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:56:29.813530] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:29.816547] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:29.816574] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:29.892032] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14758976}) [2022-08-08 10:56:30.106832] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927388271131, end_time=1659927390106804, elapsed_time=1835673, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:56:30.210850] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:30.210919] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667912, retry interval=1000000, ret=-4179) [2022-08-08 10:56:30.285920] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:30.286007] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667917, retry interval=1000000, ret=-4179) [2022-08-08 10:56:30.327000] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:30.335531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:30.524855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:30.574550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA0-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.574616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA0-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.574825] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA0-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.575014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA0-0-0] [lt=85] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.575110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.575369] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.575660] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.575797] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:30.659468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:30.725888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:30.726050] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:30.736210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:30.772441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:30.805572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:30.808653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:30.814102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:30.819194] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:30.823414] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:30.823442] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:30.845730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:30.866461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:30.871053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:30.966273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:31.031799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:31.200461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:31.204868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:31.251423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:31.263252] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31091, time_dist: generate_cluster_version=31086) [2022-08-08 10:56:31.294844] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:31.294891] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667911, retry interval=1000000, ret=-4179) [2022-08-08 10:56:31.318082] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20401) [2022-08-08 10:56:31.328301] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:31.328382] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667916, retry interval=1000000, ret=-4179) [2022-08-08 10:56:31.329919] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26232, time_dist: generate_cluster_version=26228) [2022-08-08 10:56:31.583155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA3-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.583774] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA3-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.583935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA3-0-0] [lt=99] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.585112] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EA3-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.585327] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.585483] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.585638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.585754] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:31.825876] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:31.829105] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:31.829140] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:32.191251] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927392190207 2022-08-08 02:56:32.190207|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:773061072327131542, inc:84091797, addr:{ip:"192.168.11.128", port:2882}, t:1659927392189000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5F89-0-0, start_time:1659927392190089, snapshot:1659927392107182|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927392190348|[calc_checksum_by_commit] u=20 ret:0, checksum:1010102245, checksum_log_ts:9223372036854775807|[submit_log] u=96 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=704 trans_version:1659927392190369, uref:1073741825|[exiting] u=20 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8300775, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=1006) [2022-08-08 10:56:32.317033] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:32.317073] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667910, retry interval=1000000, ret=-4179) [2022-08-08 10:56:32.354155] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:32.354246] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667915, retry interval=1000000, ret=-4179) [2022-08-08 10:56:32.650751] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:32.831365] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:32.836712] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:32.836750] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:33.339882] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:33.339932] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667909, retry interval=1000000, ret=-4179) [2022-08-08 10:56:33.377142] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:33.377204] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667914, retry interval=1000000, ret=-4179) [2022-08-08 10:56:33.681114] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=257] [dc=0] clock out of order(cur_ts=1659927393681107, cur_ts_=1659927393652698, delta=28409) [2022-08-08 10:56:33.839170] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:33.843018] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:33.843131] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:33.994443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:34.124019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.124389] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927394124385, cur_ts_=1659927394087614, delta=36771) [2022-08-08 10:56:34.144823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.165235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.295533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.338181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.340135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:34.358547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.360988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:34.370541] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:34.370752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=254] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.371705] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:34.371741] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667908, retry interval=1000000, ret=-4179) [2022-08-08 10:56:34.398445] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:34.398483] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667913, retry interval=1000000, ret=-4179) [2022-08-08 10:56:34.440332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.478923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:34.490747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.509836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.509902] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.516683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.523322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:34.541065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:34.551707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.554205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.563045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:34.583904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.585339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.611082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.612892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.611152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.611447] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927394611442, cur_ts_=1659927394588103, delta=23339) [2022-08-08 10:56:34.612199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:34.612797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.621265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.623241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.631953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.631963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.652268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.654696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.662532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.683929] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.716930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:34.728195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.735441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.735964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.747537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:34.790582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:34.790636] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.847283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:34.856576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:34.859418] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:34.862624] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:34.862718] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:34.872854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.940344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.940545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:34.947767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:34.978607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:35.024290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:35.054541] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:35.126894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:35.199179] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659927395198376 2022-08-08 02:56:35.198376|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:546987170062704260, inc:84091918, addr:{ip:"192.168.11.128", port:2882}, t:1659927395198000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5F99-0-0, start_time:1659927395198264, snapshot:1659927395127432|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927395198518|[calc_checksum_by_commit] u=21 ret:0, checksum:2916168223, checksum_log_ts:9223372036854775807|[submit_log] u=26 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=532 trans_version:1659927395198540, uref:1073741825|[exiting] u=9 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8300791, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=755) [2022-08-08 10:56:35.230465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:35.397428] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:35.397981] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=187] [dc=0] task process failed, start retry(max retry time=9223372036854667907, retry interval=1000000, ret=-4179) [2022-08-08 10:56:35.409192] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:35.419755] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=490] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:35.419811] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667912, retry interval=1000000, ret=-4179) [2022-08-08 10:56:35.643359] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:56:35.865353] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:35.870592] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:35.870634] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:36.419758] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:36.419830] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667906, retry interval=1000000, ret=-4179) [2022-08-08 10:56:36.421082] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10174, time_dist: generate_cluster_version=10172) [2022-08-08 10:56:36.440416] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:36.440468] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667911, retry interval=1000000, ret=-4179) [2022-08-08 10:56:36.478060] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:36.524212] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1479] [dc=0] clock out of order(cur_ts=1659927396524205, cur_ts_=1659927396503679, delta=20526) [2022-08-08 10:56:36.632831] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927396632822, cur_ts_=1659927396609346, delta=23476) [2022-08-08 10:56:36.659801] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=14964) [2022-08-08 10:56:36.666728] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EB2-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:36.666751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EB2-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:36.666944] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EB2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:36.667076] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EB2-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:36.667243] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:36.667974] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23863, time_dist: generate_cluster_version=23855, cluster_service_self_check=5) [2022-08-08 10:56:36.873250] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:36.876519] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:36.876557] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:37.440975] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:37.441005] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667905, retry interval=1000000, ret=-4179) [2022-08-08 10:56:37.464700] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:37.464739] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667910, retry interval=1000000, ret=-4179) [2022-08-08 10:56:37.879133] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:37.883471] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:37.883496] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:38.238032] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927398237266 2022-08-08 02:56:38.237266|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:5816956293177891994, inc:84092043, addr:{ip:"192.168.11.128", port:2882}, t:1659927398236000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABD75-0-0, start_time:1659927398237100, snapshot:1659927398185330|[end_task] u=130 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927398237414|[calc_checksum_by_commit] u=18 ret:0, checksum:3892744823, checksum_log_ts:9223372036854775807|[submit_log] u=26 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=498 trans_version:1659927398236855, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315398, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233538864|total_timeu=727) [2022-08-08 10:56:38.469117] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:38.469262] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667904, retry interval=1000000, ret=-4179) [2022-08-08 10:56:38.490095] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:38.490207] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] task process failed, start retry(max retry time=9223372036854667909, retry interval=1000000, ret=-4179) [2022-08-08 10:56:38.495206] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10132, time_dist: generate_cluster_version=10127) [2022-08-08 10:56:38.886086] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:38.889555] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:38.889579] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:39.500820] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16970, time_dist: generate_cluster_version=16967) [2022-08-08 10:56:39.524695] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12642) [2022-08-08 10:56:39.525652] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14965, time_dist: generate_cluster_version=14962) [2022-08-08 10:56:39.545087] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:39.545122] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667908, retry interval=1000000, ret=-4179) [2022-08-08 10:56:39.547111] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:39.547179] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667903, retry interval=1000000, ret=-4179) [2022-08-08 10:56:39.704192] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EBB-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:39.704232] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EBB-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:39.704459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EBB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:39.704613] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EBB-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:39.704843] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:39.706009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:39.721036] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:56:39.721074] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:56:39.810982] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927399810970, cur_ts_=1659927399785662, delta=25308) [2022-08-08 10:56:39.891466] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:39.894632] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:39.894658] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:39.929902] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14763328}) [2022-08-08 10:56:40.067161] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11686, time_dist: generate_cluster_version=11682) [2022-08-08 10:56:40.079974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:40.111774] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927398271537, end_time=1659927400111762, elapsed_time=1840225, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:56:40.350390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:40.350771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:40.544065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:40.572040] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:40.572083] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667902, retry interval=1000000, ret=-4179) [2022-08-08 10:56:40.588054] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:40.588090] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667907, retry interval=1000000, ret=-4179) [2022-08-08 10:56:40.682229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:40.741902] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:40.743334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:40.751493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:40.795282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=155] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:40.824397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:40.829881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:40.833275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:40.862187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:40.889336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:40.894365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:40.899799] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:40.903041] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:40.903076] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:40.987784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:41.052191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:41.221220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:41.223992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:41.257499] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=13] [dc=0] [trans sampling] (TRACE=begin_ts=1659927401256581 2022-08-08 02:56:41.256581|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:9067065385695964982, inc:84092153, addr:{ip:"192.168.11.128", port:2882}, t:1659927401256000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABD84-0-0, start_time:1659927401256304, snapshot:1659927401185187|[end_task] u=140 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927401256741|[calc_checksum_by_commit] u=21 ret:0, checksum:593284689, checksum_log_ts:9223372036854775807|[submit_log] u=66 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=592 trans_version:1659927401256158, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315413, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=879) [2022-08-08 10:56:41.268336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=10741] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:41.595865] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:41.595901] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667901, retry interval=1000000, ret=-4179) [2022-08-08 10:56:41.624104] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:41.624149] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667906, retry interval=1000000, ret=-4179) [2022-08-08 10:56:41.721970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EC1-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:41.722013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EC1-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:41.722222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EC1-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:41.898882] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10447, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10432) [2022-08-08 10:56:41.921958] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:41.926265] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:41.926305] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:41.976599] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=4933] [dc=0] clock out of order(cur_ts=1659927401976590, cur_ts_=1659927401953783, delta=22807) [2022-08-08 10:56:42.077670] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927402077664, cur_ts_=1659927402046381, delta=31283) [2022-08-08 10:56:42.139395] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10635, time_dist: do_cluster_heartbeat=11, generate_cluster_version=10620) [2022-08-08 10:56:42.621332] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:42.621385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667900, retry interval=1000000, ret=-4179) [2022-08-08 10:56:42.647037] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=320] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:42.648298] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=290] [dc=0] task process failed, start retry(max retry time=9223372036854667905, retry interval=1000000, ret=-4179) [2022-08-08 10:56:42.932194] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:42.965803] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=306] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:42.965829] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:43.011253] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927403011206, cur_ts_=1659927402989189, delta=22017) [2022-08-08 10:56:43.043512] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13806) [2022-08-08 10:56:43.111946] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15330) [2022-08-08 10:56:43.112151] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16404, time_dist: generate_cluster_version=16401) [2022-08-08 10:56:43.638500] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10485][0][YB42C0A80B80-0005E3CF9C184D96-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:56:43.638769] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10485][0][YB42C0A80B80-0005E3CF9C184D96-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 10:56:43.675960] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927403675950, cur_ts_=1659927403644091, delta=31859) [2022-08-08 10:56:43.677539] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=38944) [2022-08-08 10:56:43.715895] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=78168, time_dist: do_cluster_heartbeat=9, generate_cluster_version=78154) [2022-08-08 10:56:43.813654] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927403813648, cur_ts_=1659927403767393, delta=46255) [2022-08-08 10:56:43.819411] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:43.819456] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667904, retry interval=1000000, ret=-4179) [2022-08-08 10:56:43.822537] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=114180) [2022-08-08 10:56:43.822571] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=14, io_time=114206, file_id=237, flush_start_offset=50165190, file_offset=50165811) [2022-08-08 10:56:43.822661] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=24] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:7478320433638858683, inc:84092245, addr:"192.168.11.128:2882", t:1659927403689505}, log_id=8300833, used_time=118239) [2022-08-08 10:56:43.847489] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:43.847534] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667899, retry interval=1000000, ret=-4179) [2022-08-08 10:56:43.854161] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=150439, time_dist: generate_cluster_version=150435) [2022-08-08 10:56:43.887512] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=58529) [2022-08-08 10:56:43.930823] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927403930818, cur_ts_=1659927403909713, delta=21105) [2022-08-08 10:56:43.933501] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23605) [2022-08-08 10:56:43.933695] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25956, time_dist: do_cluster_heartbeat=9, generate_cluster_version=25945) [2022-08-08 10:56:44.014194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:44.014225] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:44.017483] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:44.017520] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:44.141956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.161012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.183564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.279117] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] [trans sampling] (TRACE=begin_ts=1659927404278249 2022-08-08 02:56:44.278249|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:2140691287403032159, inc:84092273, addr:{ip:"192.168.11.128", port:2882}, t:1659927404277000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABD93-0-0, start_time:1659927404278139, snapshot:1659927404208884|[end_task] u=130 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=24 stc:1659927404278405|[calc_checksum_by_commit] u=19 ret:0, checksum:141336706, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=557 trans_version:1659927404277774, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315428, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=829) [2022-08-08 10:56:44.315301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.359398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.363493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:44.380479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.382851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:44.387832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.388925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:44.465653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.498022] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10984, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10975) [2022-08-08 10:56:44.502785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:44.509520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.527077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.528321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.536098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.539060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:44.559834] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:44.571066] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.577688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.589912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:44.605811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.608418] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.626299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=1772] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.634199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.634256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.634435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=730] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:44.634535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=1769] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.639753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.644856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.651607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.655172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.672462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.676402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.681334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.702215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.733420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:44.749402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.752230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.760104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.770201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:44.806400] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=182] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:44.806626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=156] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.846053] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:44.846168] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] task process failed, start retry(max retry time=9223372036854667903, retry interval=1000000, ret=-4179) [2022-08-08 10:56:44.870127] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:44.870168] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667898, retry interval=1000000, ret=-4179) [2022-08-08 10:56:44.871528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:44.875242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:44.888178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.963648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.963680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:44.971479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:44.994896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:45.020117] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:45.025889] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:45.025962] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:45.045493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:45.070846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:45.072743] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=1419] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10751) [2022-08-08 10:56:45.143892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:45.145972] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927405145963, cur_ts_=1659927405125268, delta=20695) [2022-08-08 10:56:45.250503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:45.428404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:45.829435] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927405829427, cur_ts_=1659927405807806, delta=21621) [2022-08-08 10:56:45.872436] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48BE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:45.912317] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:45.912587] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667902, retry interval=1000000, ret=-4179) [2022-08-08 10:56:45.919266] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=762] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13744) [2022-08-08 10:56:45.920345] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17050, time_dist: do_cluster_heartbeat=9, generate_cluster_version=17039) [2022-08-08 10:56:45.930448] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:45.930501] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667897, retry interval=1000000, ret=-4179) [2022-08-08 10:56:46.028982] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:46.036962] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:46.037479] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=503] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:46.545123] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:46.773289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ED0-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.773464] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ED0-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.773722] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ED0-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.773873] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ED0-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.774039] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.774235] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.774382] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.774615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:46.887583] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927406887577, cur_ts_=1659927406860257, delta=27320) [2022-08-08 10:56:46.956740] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14033, time_dist: generate_cluster_version=14031) [2022-08-08 10:56:46.962297] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:46.962348] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667896, retry interval=1000000, ret=-4179) [2022-08-08 10:56:46.969420] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:46.969483] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667901, retry interval=1000000, ret=-4179) [2022-08-08 10:56:47.042861] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:47.047373] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:47.047411] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:47.112723] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927407112717, cur_ts_=1659927407083782, delta=28935) [2022-08-08 10:56:47.292335] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=5] [dc=0] [trans sampling] (TRACE=begin_ts=1659927407291390 2022-08-08 02:56:47.291390|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:303255655370402209, inc:84092390, addr:{ip:"192.168.11.128", port:2882}, t:1659927407290000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A5FD5-0-0, start_time:1659927407291261, snapshot:1659927407235857|[end_task] u=151 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927407291559|[calc_checksum_by_commit] u=20 ret:0, checksum:2419717732, checksum_log_ts:9223372036854775807|[submit_log] u=77 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=574 trans_version:1659927407291580, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8300851, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=879) [2022-08-08 10:56:47.830299] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927407830291, cur_ts_=1659927407796405, delta=33886) [2022-08-08 10:56:47.894975] TRACE [TRACE]ob_trans_ctx.cpp:485 [10632][0][YB42C0A80B80-0005E3CF9BC835F8-0-0] [lt=22] [dc=0] [slow trans] (TRACE=begin_ts=1659927390061863 2022-08-08 02:56:30.061863|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:3388301855247285742, inc:84091721, addr:{ip:"192.168.11.128", port:2882}, t:1659927390060023}, pkey:{tid:1100611139453778, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835F6-0-0, start_time:1659927390061580, snapshot:1659927389999075|[end_task] u=134 ret:0, is_rollback:false, uref:1073741825|[handle_timeout] u=10288826 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=209209 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[handle_timeout] u=410173 ret:0, used:{click:[1, 1, 6]}, uref:1073741826|[handle_timeout] u=809460 ret:0, used:{click:[2, 0, 7]}, uref:1073741826|[handle_timeout] u=1601727 ret:0, used:{click:[1, 0, 7]}, uref:1073741826|[start_task] u=816349 ret:0, sql_no:8589934594, uref:1073741825, trace_id:YB42C0A80B80-0005E3CF9BC835F7-0-0, start_time:1659927404197520, snapshot:1659927404108565|[end_task] u=159 ret:0, is_rollback:false, uref:1073741825|[handle_timeout] u=2190122 ret:0, used:{click:[1, 0, 6]}, uref:1073741826|[set_stc] u=1481364 stc:1659927407869393|[calc_checksum_by_commit] u=30 ret:0, checksum:1930962082, checksum_log_ts:9223372036854775807|[submit_log] u=2405 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=4 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=529 trans_version:1659927407869426, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:3221573953, arg2:0|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:6, logging:0, pending:0, uref:1|[end_trans_cb] u=22527 ret:0, param:0, param:true|[destroy] u=7 arg1:140085233547952|total_timeu=17833056) [2022-08-08 10:56:47.895017] WARN [CLOG] submit_log_succ_cb (ob_log_task.cpp:323) [10632][0][YB42C0A80B80-0005E3CF9BC835F8-0-0] [lt=30] [dc=0] transaction on success costs(on_success_duration=22656, pkey={tid:1100611139453778, partition_id:0, part_cnt:0}, log_id=6) [2022-08-08 10:56:47.895127] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10632][0][YB42C0A80B80-0005E3CF9BC835F8-0-0] [lt=7] [dc=0] flush_cb cost too much time(partition_key_={tid:1100611139453778, partition_id:0, part_cnt:0}, arg={log_type:202, log_id:6, proposal_id:{time_to_usec:1659926840155590, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:50206242, size:668}, after_consume_timestamp:1659927407871840}, end_time=1659927407894228, begin_time=1659927407871840, cost_time=22388) [2022-08-08 10:56:47.971254] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.971291] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.971315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.971356] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.971535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.971587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.972135] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48C1-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:47.995992] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:47.996099] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667900, retry interval=1000000, ret=-4179) [2022-08-08 10:56:47.998549] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:47.998614] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667895, retry interval=1000000, ret=-4179) [2022-08-08 10:56:48.050350] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:48.056064] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:48.056103] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:48.336911] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11203) [2022-08-08 10:56:48.337193] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=182] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12485, time_dist: generate_cluster_version=12482) [2022-08-08 10:56:48.356641] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19685) [2022-08-08 10:56:48.358418] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=33943, time_dist: generate_cluster_version=33939) [2022-08-08 10:56:48.791560] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10270, time_dist: generate_cluster_version=10265) [2022-08-08 10:56:48.792861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ED6-0-0] [lt=49] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:48.793165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2ED6-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:48.793463] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:48.793769] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:48.794153] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:48.794221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:49.029642] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:49.029680] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667899, retry interval=1000000, ret=-4179) [2022-08-08 10:56:49.034508] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:49.034942] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667894, retry interval=1000000, ret=-4179) [2022-08-08 10:56:49.058878] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:49.061890] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:49.061927] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:49.353076] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10703) [2022-08-08 10:56:49.353289] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12106, time_dist: generate_cluster_version=12103) [2022-08-08 10:56:49.375446] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11878) [2022-08-08 10:56:49.375677] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13159, time_dist: generate_cluster_version=13156) [2022-08-08 10:56:49.546964] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11131) [2022-08-08 10:56:49.547193] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12101, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12089) [2022-08-08 10:56:49.582204] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15204) [2022-08-08 10:56:49.582494] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16542, time_dist: print_stat=11, generate_cluster_version=16528) [2022-08-08 10:56:49.716066] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11168) [2022-08-08 10:56:49.716321] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12235, time_dist: generate_cluster_version=12233) [2022-08-08 10:56:49.721672] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:56:49.721712] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:56:49.756215] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11940) [2022-08-08 10:56:49.756781] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13369, time_dist: generate_cluster_version=13366) [2022-08-08 10:56:49.800417] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:49.800444] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:49.800455] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:49.800619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:49.892149] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927409892142, cur_ts_=1659927409870734, delta=21408) [2022-08-08 10:56:49.893629] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14767712}) [2022-08-08 10:56:49.943206] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12189) [2022-08-08 10:56:49.943452] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13167, time_dist: generate_cluster_version=13165) [2022-08-08 10:56:49.990967] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10696) [2022-08-08 10:56:49.991394] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12383, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12370) [2022-08-08 10:56:50.064016] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:50.069213] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:50.069237] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:50.086601] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:50.086655] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667898, retry interval=1000000, ret=-4179) [2022-08-08 10:56:50.095704] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927408271766, end_time=1659927410095690, elapsed_time=1823924, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:56:50.098219] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:50.098270] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667893, retry interval=1000000, ret=-4179) [2022-08-08 10:56:50.101958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:50.362063] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835F8-0-0] [lt=25] [dc=0] [trans sampling] (TRACE=begin_ts=1659927410361223 2022-08-08 02:56:50.361223|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:15881408031710267469, inc:84092504, addr:{ip:"192.168.11.128", port:2882}, t:1659927410359000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=56 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABDB1-0-0, start_time:1659927410361114, snapshot:1659927410271965|[end_task] u=143 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927410361433|[calc_checksum_by_commit] u=19 ret:0, checksum:1867712164, checksum_log_ts:9223372036854775807|[submit_log] u=42 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=491 trans_version:1659927410361364, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315458, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=800) [2022-08-08 10:56:50.370266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:50.370292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:50.561690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:50.706059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:50.757391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:50.767684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:50.767875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:50.814154] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:50.845626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:50.845941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:50.856780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:50.887610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:50.908705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:50.918999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:51.011693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:51.072552] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:51.075825] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:51.075891] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:51.077147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:51.091948] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:51.091983] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:51.120565] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:51.120598] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667897, retry interval=1000000, ret=-4179) [2022-08-08 10:56:51.123970] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19754) [2022-08-08 10:56:51.124925] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26983, time_dist: do_cluster_heartbeat=8, generate_cluster_version=26971) [2022-08-08 10:56:51.128117] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:51.128177] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667892, retry interval=1000000, ret=-4179) [2022-08-08 10:56:51.243340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:51.243341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:51.297874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:51.299119] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14738, time_dist: do_cluster_heartbeat=9, generate_cluster_version=14725) [2022-08-08 10:56:51.816115] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:51.816233] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:51.816366] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:51.816439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:52.078482] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:52.082042] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:52.082077] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:52.131698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.132217] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.132596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=198] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.132695] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=74] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.132747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.133097] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.133113] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.133125] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.133174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4906-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:52.159545] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:52.159579] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667891, retry interval=1000000, ret=-4179) [2022-08-08 10:56:52.160471] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:52.160495] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854667896, retry interval=1000000, ret=-4179) [2022-08-08 10:56:52.262431] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659927412262405, cur_ts_=1659927412239143, delta=23262) [2022-08-08 10:56:52.335283] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12718, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12704) [2022-08-08 10:56:52.441496] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927412441489, cur_ts_=1659927412419230, delta=22259) [2022-08-08 10:56:52.675483] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927412675476, cur_ts_=1659927412642397, delta=33079) [2022-08-08 10:56:52.675752] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:52.824005] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:52.824035] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:52.824050] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:53.086955] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:53.090512] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:53.090549] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:53.199442] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:53.199632] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] task process failed, start retry(max retry time=9223372036854667890, retry interval=1000000, ret=-4179) [2022-08-08 10:56:53.201457] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:53.201488] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667895, retry interval=1000000, ret=-4179) [2022-08-08 10:56:53.830922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EE5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:53.830988] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EE5-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:53.831158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EE5-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:53.831357] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EE5-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:53.831525] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:54.034218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:54.093257] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:54.096813] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=156] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:54.096852] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:54.158910] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.183525] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.204162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:54.204181] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:54.204484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=171] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.228269] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:54.228305] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667894, retry interval=1000000, ret=-4179) [2022-08-08 10:56:54.232523] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:54.232631] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667889, retry interval=1000000, ret=-4179) [2022-08-08 10:56:54.330806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.375443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.383422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:54.398402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:54.401330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.409685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:54.412691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.486451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.518588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:54.528789] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.550063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.551053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.559941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=135] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.560652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:54.575890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:54.593696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.601060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.613290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:56:54.623655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.623656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.644342] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.652820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.654702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.654756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:56:54.658726] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.659751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.664333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.674552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.674579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.691930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.696475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.699982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.722691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.751040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:54.772179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.774284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.782202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.792091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:56:54.828795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:54.831330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.886709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:56:54.897082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:54.907282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.980717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.982083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:54.989728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:55.019771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:55.065476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:56:55.093338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:55.102580] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:55.106344] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:55.106386] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:55.166844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=1511] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:56:55.265827] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:55.265824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:55.265861] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667888, retry interval=1000000, ret=-4179) [2022-08-08 10:56:55.268621] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:55.268674] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667893, retry interval=1000000, ret=-4179) [2022-08-08 10:56:55.449318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:56:55.674027] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:56:55.845276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EEB-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:55.845331] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EEB-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:55.845624] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EEB-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:55.845725] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:55.846645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:56.108999] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:56.112342] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:56.112394] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:56.297433] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:56.297481] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667892, retry interval=1000000, ret=-4179) [2022-08-08 10:56:56.307477] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:56.307529] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667887, retry interval=1000000, ret=-4179) [2022-08-08 10:56:56.535852] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927416535846, cur_ts_=1659927416507949, delta=27897) [2022-08-08 10:56:56.604327] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:57.115000] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:57.119301] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:57.119370] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:57.309491] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A51-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:57.309560] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A51-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:57.330439] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:57.330478] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667891, retry interval=1000000, ret=-4179) [2022-08-08 10:56:57.336436] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:57.336469] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667886, retry interval=1000000, ret=-4179) [2022-08-08 10:56:57.869204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:57.869340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=64] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:57.869350] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:57.869355] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:57.869570] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:57.869593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:57.869819] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EF1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:58.121743] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:58.124847] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:58.124887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:58.333225] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C490F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:58.333280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C490F-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:58.333656] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C490F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:58.338368] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48D0-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:58.338605] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:58.369662] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:58.369715] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667890, retry interval=1000000, ret=-4179) [2022-08-08 10:56:58.383245] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:58.383311] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667885, retry interval=1000000, ret=-4179) [2022-08-08 10:56:58.574466] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19411, time_dist: generate_cluster_version=19407) [2022-08-08 10:56:58.574969] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18579, time_dist: generate_cluster_version=18577) [2022-08-08 10:56:58.807767] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17305) [2022-08-08 10:56:58.807953] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18359, time_dist: generate_cluster_version=18357) [2022-08-08 10:56:58.919690] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927418919681, cur_ts_=1659927418877447, delta=42234) [2022-08-08 10:56:58.970790] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13724) [2022-08-08 10:56:58.971025] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15133, time_dist: do_cluster_heartbeat=7, generate_cluster_version=15123) [2022-08-08 10:56:59.130739] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:56:59.135720] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:56:59.135770] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:56:59.375661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:59.375713] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:56:59.405330] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:59.405519] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] task process failed, start retry(max retry time=9223372036854667889, retry interval=1000000, ret=-4179) [2022-08-08 10:56:59.419388] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=164] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:56:59.419631] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] task process failed, start retry(max retry time=9223372036854667884, retry interval=1000000, ret=-4179) [2022-08-08 10:56:59.425765] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:56:59.583021] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927419583014, cur_ts_=1659927419550715, delta=32299) [2022-08-08 10:56:59.586732] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927419583396 2022-08-08 02:56:59.583396|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:18367909364515368423, inc:84092851, addr:{ip:"192.168.11.128", port:2882}, t:1659927419550000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABDDF-0-0, start_time:1659927419583285, snapshot:1659927419583109|[end_task] u=167 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927419583581|[calc_checksum_by_commit] u=20 ret:0, checksum:52268124, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3032 trans_version:1659927419583110, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8315504, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=3297) [2022-08-08 10:56:59.587115] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=339] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511628002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8315504, proposal_id:{time_to_usec:1659815620290075, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:50316158, size:568}, after_consume_timestamp:1659927419550715}, end_time=1659927419585923, begin_time=1659927419550715, cost_time=35208) [2022-08-08 10:56:59.732941] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:56:59.732960] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:56:59.777891] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927419777885, cur_ts_=1659927419746675, delta=31210) [2022-08-08 10:56:59.778188] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25003) [2022-08-08 10:56:59.778434] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26066, time_dist: do_cluster_heartbeat=7, generate_cluster_version=26057) [2022-08-08 10:56:59.823085] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10106) [2022-08-08 10:56:59.823331] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11210, time_dist: generate_cluster_version=11207) [2022-08-08 10:56:59.919626] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14772032}) [2022-08-08 10:56:59.933240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:59.933471] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:56:59.933540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:00.008837] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18010) [2022-08-08 10:57:00.009276] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19214, time_dist: generate_cluster_version=19211) [2022-08-08 10:57:00.051780] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10645) [2022-08-08 10:57:00.054783] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14660, time_dist: generate_cluster_version=14655) [2022-08-08 10:57:00.115804] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927418272701, end_time=1659927420115793, elapsed_time=1843092, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:57:00.124278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:00.137561] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:00.140962] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:00.140999] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:00.356605] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20152) [2022-08-08 10:57:00.356859] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=318] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21154, time_dist: do_cluster_heartbeat=9, generate_cluster_version=21143) [2022-08-08 10:57:00.388031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:00.394509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:00.422517] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48D3-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:00.422540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48D3-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:00.422552] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C48D3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:00.436171] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:00.436207] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667888, retry interval=1000000, ret=-4179) [2022-08-08 10:57:00.441730] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:00.441762] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667883, retry interval=1000000, ret=-4179) [2022-08-08 10:57:00.581858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:00.723691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:00.724292] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927420724286, cur_ts_=1659927420702167, delta=22119) [2022-08-08 10:57:00.780661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:00.790904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:00.791081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:00.832545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:00.862853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:00.863769] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:00.874272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:00.905529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:00.926282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:00.938162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:00.946746] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EFA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:00.946803] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EFA-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:00.946811] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EFA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:00.946822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EFA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:00.946830] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2EFA-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:01.034842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:01.093390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:01.144848] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:01.149494] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:01.149535] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:01.265064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:01.267626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:01.318136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:01.441196] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:01.441271] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:01.480844] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] clock out of order(cur_ts=1659927421480838, cur_ts_=1659927421452368, delta=28470) [2022-08-08 10:57:01.500217] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:01.530754] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] task process failed, start retry(max retry time=9223372036854667882, retry interval=1000000, ret=-4179) [2022-08-08 10:57:01.535010] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:01.535047] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667887, retry interval=1000000, ret=-4179) [2022-08-08 10:57:01.900563] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927421900552, cur_ts_=1659927421880167, delta=20385) [2022-08-08 10:57:02.151953] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:02.155475] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:02.155500] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:02.364295] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927422364286, cur_ts_=1659927422336936, delta=27350) [2022-08-08 10:57:02.558783] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:02.558821] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667881, retry interval=1000000, ret=-4179) [2022-08-08 10:57:02.560051] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835F8-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927422559251 2022-08-08 02:57:02.559251|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:4828011613530574391, inc:84092963, addr:{ip:"192.168.11.128", port:2882}, t:1659927422557000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=4 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7CA2-0-0, start_time:1659927422559143, snapshot:1659927422505169|[end_task] u=114 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=7 stc:1659927422559380|[calc_checksum_by_commit] u=17 ret:0, checksum:3621216730, checksum_log_ts:9223372036854775807|[submit_log] u=42 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=539 trans_version:1659927422557651, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:987755, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=763) [2022-08-08 10:57:02.560739] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:02.560797] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667886, retry interval=1000000, ret=-4179) [2022-08-08 10:57:02.804526] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13629, time_dist: do_cluster_heartbeat=11, generate_cluster_version=13614) [2022-08-08 10:57:03.157864] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:03.161109] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:03.161151] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:03.587096] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:03.587220] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] task process failed, start retry(max retry time=9223372036854667880, retry interval=1000000, ret=-4179) [2022-08-08 10:57:03.587833] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:03.587860] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667885, retry interval=1000000, ret=-4179) [2022-08-08 10:57:03.968155] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927423968149, cur_ts_=1659927423943285, delta=24864) [2022-08-08 10:57:03.971318] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F03-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:03.971367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F03-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:03.971384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F03-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:03.971446] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F03-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:03.972190] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:04.060202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:04.163943] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:04.170494] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:04.170542] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:04.176991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.208870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.224123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.354005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.394407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.408790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:04.415012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:04.419240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.429644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:04.436052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.506070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.541699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:04.553330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.567518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.568015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.578025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.583423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:04.598991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:04.609228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.620444] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:04.620484] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667884, retry interval=1000000, ret=-4179) [2022-08-08 10:57:04.623297] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:04.623328] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667879, retry interval=1000000, ret=-4179) [2022-08-08 10:57:04.623353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.633484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:04.646666] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.648095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.662198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.671235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.677900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:04.678394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.679002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.679160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.689011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.696635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.698491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.712915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.717014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.720758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.745088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.767854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:04.793164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.799621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.800172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.810029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:04.852205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:04.856165] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.905876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:04.917785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:04.930132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:04.999022] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:05.003456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:05.012379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:05.041742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:05.083357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:05.112071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:05.173616] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:05.177068] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:05.177112] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:05.184473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:05.285276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:05.472745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:05.602694] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927425601144 2022-08-08 02:57:05.601144|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:7200177191829098080, inc:84093074, addr:{ip:"192.168.11.128", port:2882}, t:1659927425599000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABDFE-0-0, start_time:1659927425601030, snapshot:1659927425527832|[end_task] u=127 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927425601288|[calc_checksum_by_commit] u=34 ret:0, checksum:4122618097, checksum_log_ts:9223372036854775807|[submit_log] u=44 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1241 trans_version:1659927425599668, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8315535, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=1511) [2022-08-08 10:57:05.625503] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625548] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625581] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625592] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625847] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.625907] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC60E2-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:05.651296] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:05.651464] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] task process failed, start retry(max retry time=9223372036854667883, retry interval=1000000, ret=-4179) [2022-08-08 10:57:05.653428] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:05.653472] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667878, retry interval=1000000, ret=-4179) [2022-08-08 10:57:05.984941] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:05.984968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:05.985082] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:06.180751] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:06.184291] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:06.184328] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:06.656496] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19651) [2022-08-08 10:57:06.656644] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20525, time_dist: generate_cluster_version=20523) [2022-08-08 10:57:06.675395] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:06.680779] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:06.680818] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667882, retry interval=1000000, ret=-4179) [2022-08-08 10:57:06.691653] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14190) [2022-08-08 10:57:06.698354] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=166] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:06.698423] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667877, retry interval=1000000, ret=-4179) [2022-08-08 10:57:07.187067] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:07.190481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:07.190540] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:07.704611] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:07.704650] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667881, retry interval=1000000, ret=-4179) [2022-08-08 10:57:07.720134] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:07.720168] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667876, retry interval=1000000, ret=-4179) [2022-08-08 10:57:07.720266] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14938) [2022-08-08 10:57:07.979069] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927427979057, cur_ts_=1659927427946462, delta=32595) [2022-08-08 10:57:08.192983] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:08.198102] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:08.198141] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:08.390212] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927428390206, cur_ts_=1659927428368051, delta=22155) [2022-08-08 10:57:08.615118] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] [trans sampling] (TRACE=begin_ts=1659927428613947 2022-08-08 02:57:08.613947|[init] u=0 arg1:140085233537488, ctx_type:"participant", trans_type:0, trans_id:{hash:10030348205585369325, inc:84093196, addr:{ip:"192.168.11.128", port:2882}, t:1659927428612000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABE0D-0-0, start_time:1659927428613830, snapshot:1659927428540485|[end_task] u=165 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927428614130|[calc_checksum_by_commit] u=20 ret:0, checksum:3201553772, checksum_log_ts:9223372036854775807|[submit_log] u=28 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=801 trans_version:1659927428612636, uref:1073741825|[exiting] u=70 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8315550, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233538864|total_timeu=1130) [2022-08-08 10:57:08.730121] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:08.730400] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667880, retry interval=1000000, ret=-4179) [2022-08-08 10:57:08.743916] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:08.744113] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=158] [dc=0] task process failed, start retry(max retry time=9223372036854667875, retry interval=1000000, ret=-4179) [2022-08-08 10:57:08.744258] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12587) [2022-08-08 10:57:08.862969] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1002] [dc=0] clock out of order(cur_ts=1659927428862962, cur_ts_=1659927428841545, delta=21417) [2022-08-08 10:57:08.899350] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35391) [2022-08-08 10:57:08.899679] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=36486, time_dist: do_cluster_heartbeat=8, generate_cluster_version=36475) [2022-08-08 10:57:09.200890] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:09.203977] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:09.204001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:09.285194] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927429285190, cur_ts_=1659927429252057, delta=33133) [2022-08-08 10:57:09.466194] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14251, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14236) [2022-08-08 10:57:09.690817] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10920) [2022-08-08 10:57:09.691051] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=151] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13566, time_dist: generate_cluster_version=13564) [2022-08-08 10:57:09.691549] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12821, time_dist: generate_cluster_version=12819) [2022-08-08 10:57:09.727518] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:57:09.727572] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:57:09.789733] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927429789722, cur_ts_=1659927429768975, delta=20747) [2022-08-08 10:57:09.802261] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:09.802313] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667874, retry interval=1000000, ret=-4179) [2022-08-08 10:57:09.808103] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:09.808155] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667879, retry interval=1000000, ret=-4179) [2022-08-08 10:57:09.897511] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14776448}) [2022-08-08 10:57:10.033063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:10.033176] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:10.033218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:10.033875] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:10.094426] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927428274064, end_time=1659927430094411, elapsed_time=1820347, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:57:10.205736] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:10.208930] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:10.208994] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:10.409683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:10.410786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:10.602055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:10.739828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:10.805297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:10.812592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:10.812892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:10.830960] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:10.830991] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667873, retry interval=1000000, ret=-4179) [2022-08-08 10:57:10.833779] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:10.833826] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667878, retry interval=1000000, ret=-4179) [2022-08-08 10:57:10.854582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:10.883795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:10.886738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:10.898246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:10.921466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:10.941494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:10.953763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:11.054531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:11.116483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:11.212722] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:11.216809] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:11.216836] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:11.283589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:11.285552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:11.339139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:11.644759] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927431643899 2022-08-08 02:57:11.643899|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:259204419015552778, inc:84093306, addr:{ip:"192.168.11.128", port:2882}, t:1659927431643000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABE1C-0-0, start_time:1659927431643789, snapshot:1659927431566334|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927431644040|[calc_checksum_by_commit] u=19 ret:0, checksum:3794134749, checksum_log_ts:9223372036854775807|[submit_log] u=70 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=542 trans_version:1659927431643643, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315565, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=820) [2022-08-08 10:57:11.855823] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:11.855875] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667872, retry interval=1000000, ret=-4179) [2022-08-08 10:57:11.875101] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:11.875305] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=135] [dc=0] task process failed, start retry(max retry time=9223372036854667877, retry interval=1000000, ret=-4179) [2022-08-08 10:57:12.055921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F1B-0-0] [lt=81] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.055951] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F1B-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.056092] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F1B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.056221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F1B-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.056384] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.056552] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.056652] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.056760] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:12.219571] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:12.223318] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:12.223364] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:12.676009] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:12.882765] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:12.882800] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667871, retry interval=1000000, ret=-4179) [2022-08-08 10:57:12.898092] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:12.898141] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667876, retry interval=1000000, ret=-4179) [2022-08-08 10:57:13.206183] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927433206172, cur_ts_=1659927433185220, delta=20952) [2022-08-08 10:57:13.228877] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:13.232109] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:13.232162] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:13.441997] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927433441990, cur_ts_=1659927433410402, delta=31588) [2022-08-08 10:57:13.910068] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:13.910106] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667870, retry interval=1000000, ret=-4179) [2022-08-08 10:57:13.923094] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:13.923518] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667875, retry interval=1000000, ret=-4179) [2022-08-08 10:57:14.070100] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:14.070166] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:14.081051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:14.204348] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.247317] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:14.248536] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927434248530, cur_ts_=1659927434221848, delta=26682) [2022-08-08 10:57:14.248556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.248678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.256474] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=1769] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:14.257147] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=658] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:14.377669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.427593] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:14.432387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:14.427750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.430442] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=573] [dc=0] clock out of order(cur_ts=1659927434430432, cur_ts_=1659927434399936, delta=30496) [2022-08-08 10:57:14.442660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.454121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=564] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:14.458090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.531138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.562642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:14.577277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.587504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.587613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.593523] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.606857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:14.621155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:14.629111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.647144] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.651057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:14.665064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.671823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.685698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.693244] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659927434692308 2022-08-08 02:57:14.692308|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:2057522345901455568, inc:84093418, addr:{ip:"192.168.11.128", port:2882}, t:1659927434691000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A605D-0-0, start_time:1659927434692188, snapshot:1659927434680925|[end_task] u=122 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927434692448|[calc_checksum_by_commit] u=20 ret:0, checksum:4255319220, checksum_log_ts:9223372036854775807|[submit_log] u=73 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=599 trans_version:1659927434692469, uref:1073741825|[exiting] u=24 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8300987, logging:0, pending:0, uref:1|[end_trans_cb] u=30 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=896) [2022-08-08 10:57:14.696153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.698681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:14.699691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.700131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.718114] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.718150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.718273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.718459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.728570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.744008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.744172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.761071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.790867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:14.811869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.817613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.822204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.832465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:14.871708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.871810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:14.921298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:14.939422] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:14.939476] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667869, retry interval=1000000, ret=-4179) [2022-08-08 10:57:14.940850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:14.949167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:14.949608] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:14.949714] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] task process failed, start retry(max retry time=9223372036854667874, retry interval=1000000, ret=-4179) [2022-08-08 10:57:15.022155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:15.023772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:15.033605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:15.064663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:15.101352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:15.128583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:15.201085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:15.260013] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:15.263487] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:15.263509] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:15.300723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:15.496752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:15.699631] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:57:15.951636] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A6C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:15.969216] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:15.969431] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] task process failed, start retry(max retry time=9223372036854667868, retry interval=1000000, ret=-4179) [2022-08-08 10:57:15.971686] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:15.971720] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667873, retry interval=1000000, ret=-4179) [2022-08-08 10:57:16.266656] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:16.270032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:16.270055] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:16.762926] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:16.857618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:16.998596] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:16.998652] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667867, retry interval=1000000, ret=-4179) [2022-08-08 10:57:17.001588] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:17.001648] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667872, retry interval=1000000, ret=-4179) [2022-08-08 10:57:17.274099] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:17.277290] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:17.277345] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:17.825005] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2598] [dc=0] clock out of order(cur_ts=1659927437824996, cur_ts_=1659927437704851, delta=120145) [2022-08-08 10:57:17.879352] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=114181) [2022-08-08 10:57:17.879402] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=28, io_time=114209, file_id=237, flush_start_offset=50486030, file_offset=50486652) [2022-08-08 10:57:17.879471] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:14899219340744276263, inc:84093546, addr:"192.168.11.128:2882", t:1659927437704000}, log_id=8301002, used_time=149566) [2022-08-08 10:57:17.879571] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] [trans sampling] (TRACE=begin_ts=1659927437729582 2022-08-08 02:57:17.729582|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:14899219340744276263, inc:84093546, addr:{ip:"192.168.11.128", port:2882}, t:1659927437704000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A606C-0-0, start_time:1659927437729419, snapshot:1659927437694699|[end_task] u=218 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927437729822|[calc_checksum_by_commit] u=25 ret:0, checksum:3748050399, checksum_log_ts:9223372036854775807|[submit_log] u=38 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=149603 trans_version:1659927437729849, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301002, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=149938) [2022-08-08 10:57:17.879661] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=150682, time_dist: generate_cluster_version=150679) [2022-08-08 10:57:17.933167] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927437933161, cur_ts_=1659927437895316, delta=37845) [2022-08-08 10:57:17.933853] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=54427) [2022-08-08 10:57:17.933979] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=20] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:15722242836745263209, inc:84093547, addr:"192.168.11.128:2882", t:1659927437704851}, log_id=8315595, used_time=172154) [2022-08-08 10:57:17.934666] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=185303, time_dist: generate_cluster_version=185192) [2022-08-08 10:57:18.065667] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927438065661, cur_ts_=1659927437978791, delta=86870) [2022-08-08 10:57:18.073832] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=139938) [2022-08-08 10:57:18.074044] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] slow flush(flush_time=36, io_time=140110, file_id=237, flush_start_offset=50487269, file_offset=50487890) [2022-08-08 10:57:18.077254] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=24] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:1819534823538306916, inc:84093551, addr:"192.168.11.128:2882", t:1659927437895316}, log_id=8301003, used_time=144096) [2022-08-08 10:57:18.078091] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=146647, time_dist: do_cluster_heartbeat=12, generate_cluster_version=146632) [2022-08-08 10:57:18.140901] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=66832) [2022-08-08 10:57:18.143619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:18.143633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:18.148509] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=55] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:2514038267484568432, inc:84093554, addr:"192.168.11.128:2882", t:1659927437978791}, log_id=8315596, used_time=147095) [2022-08-08 10:57:18.148671] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=156800, time_dist: do_cluster_heartbeat=11, generate_cluster_version=156787) [2022-08-08 10:57:18.190972] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:18.191010] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667866, retry interval=1000000, ret=-4179) [2022-08-08 10:57:18.192729] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:18.192760] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667871, retry interval=1000000, ret=-4179) [2022-08-08 10:57:18.288744] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:18.293836] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:18.294007] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=157] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:18.384972] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19993) [2022-08-08 10:57:18.385808] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21588, time_dist: do_cluster_heartbeat=8, generate_cluster_version=21577) [2022-08-08 10:57:18.397734] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12719) [2022-08-08 10:57:18.397994] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22426, time_dist: generate_cluster_version=22422) [2022-08-08 10:57:18.437037] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927438437029, cur_ts_=1659927438410581, delta=26448) [2022-08-08 10:57:18.574015] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12545, time_dist: generate_cluster_version=12543) [2022-08-08 10:57:18.727355] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927438727347, cur_ts_=1659927438685489, delta=41858) [2022-08-08 10:57:18.844574] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927438844568, cur_ts_=1659927438773461, delta=71107) [2022-08-08 10:57:18.952716] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927438952709, cur_ts_=1659927438911998, delta=40711) [2022-08-08 10:57:19.184902] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927439184896, cur_ts_=1659927439148849, delta=36047) [2022-08-08 10:57:19.196706] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11676) [2022-08-08 10:57:19.198011] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13967, time_dist: do_cluster_heartbeat=7, generate_cluster_version=13957) [2022-08-08 10:57:19.241866] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:19.241907] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667865, retry interval=1000000, ret=-4179) [2022-08-08 10:57:19.247093] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:19.247127] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667870, retry interval=1000000, ret=-4179) [2022-08-08 10:57:19.298314] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:19.303738] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:19.303767] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:19.440533] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] clock out of order(cur_ts=1659927439440524, cur_ts_=1659927439418190, delta=22334) [2022-08-08 10:57:19.722291] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:57:19.722321] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:57:19.767548] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927439767540, cur_ts_=1659927439740569, delta=26971) [2022-08-08 10:57:20.251229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:20.251246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:20.294828] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:20.294879] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667864, retry interval=1000000, ret=-4179) [2022-08-08 10:57:20.295600] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:20.295632] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667869, retry interval=1000000, ret=-4179) [2022-08-08 10:57:20.305818] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:20.308931] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:20.308973] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:20.326171] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14780832}) [2022-08-08 10:57:20.434271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:20.435216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:20.512143] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927438278633, end_time=1659927440512132, elapsed_time=2233499, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:57:20.624923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:20.759454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:20.784082] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927440779300 2022-08-08 02:57:20.779300|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:10059768656738057622, inc:84093653, addr:{ip:"192.168.11.128", port:2882}, t:1659927440778000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABE48-0-0, start_time:1659927440779187, snapshot:1659927440769201|[end_task] u=126 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927440779444|[calc_checksum_by_commit] u=20 ret:0, checksum:4240063032, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=4470 trans_version:1659927440778618, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315609, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=4702) [2022-08-08 10:57:20.820981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:20.830235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:20.831219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:20.873616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:20.904195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:20.904288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:20.916400] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:20.937414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:20.962121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:20.988835] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927440988829, cur_ts_=1659927440965539, delta=23290) [2022-08-08 10:57:20.989615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:21.072121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:21.134248] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F36-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:21.134367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F36-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:21.134971] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F36-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:21.135210] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:21.135527] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:21.137309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:21.306572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:21.306987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:21.320730] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:21.326455] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:21.326473] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:21.329035] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:21.329144] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667868, retry interval=1000000, ret=-4179) [2022-08-08 10:57:21.334419] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:21.334464] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667863, retry interval=1000000, ret=-4179) [2022-08-08 10:57:21.359981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:22.142119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:22.142360] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:22.142525] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:22.142687] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:22.330114] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:22.333478] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:22.333501] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:22.337982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:22.338028] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:22.359256] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:22.359385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667867, retry interval=1000000, ret=-4179) [2022-08-08 10:57:22.371981] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:22.372013] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667862, retry interval=1000000, ret=-4179) [2022-08-08 10:57:22.372101] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12090) [2022-08-08 10:57:23.340144] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:23.345965] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:23.346024] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:23.398781] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:23.398812] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667866, retry interval=1000000, ret=-4179) [2022-08-08 10:57:23.401888] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:23.401936] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667861, retry interval=1000000, ret=-4179) [2022-08-08 10:57:23.801806] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] [trans sampling] (TRACE=begin_ts=1659927443800832 2022-08-08 02:57:23.800832|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:5774449999854275080, inc:84093769, addr:{ip:"192.168.11.128", port:2882}, t:1659927443800000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=37 ret:0, left_time:10000000, uref:1073741825|[start_task] u=10 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A608A-0-0, start_time:1659927443800691, snapshot:1659927443799200|[end_task] u=159 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927443801044|[calc_checksum_by_commit] u=22 ret:0, checksum:3293440430, checksum_log_ts:9223372036854775807|[submit_log] u=49 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=602 trans_version:1659927443801067, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8301032, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=928) [2022-08-08 10:57:24.103202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:24.158079] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F3F-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:24.158242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F3F-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:24.158303] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F3F-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:24.222189] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=169] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10183, time_dist: generate_cluster_version=10179) [2022-08-08 10:57:24.227961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.270422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=631] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.272979] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=709] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.348535] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:24.353370] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:24.353417] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:24.399991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.440018] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:24.440093] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667860, retry interval=1000000, ret=-4179) [2022-08-08 10:57:24.442896] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:24.442930] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667865, retry interval=1000000, ret=-4179) [2022-08-08 10:57:24.448754] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=5203] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.448851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:24.450044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=4308] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:24.455451] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835FE-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:24.455487] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC835FE-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:24.468454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.472969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:24.478254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.554322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.585780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:24.592561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.603441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.607960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.610926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.625740] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:24.637667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:24.644800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.668891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.673890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:24.685841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.691254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.705222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.719100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.721999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.721999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:24.732244] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.736419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.736520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.738386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.743078] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.753351] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.765175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.765620] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.776554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.809434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:24.830600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.836060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.842440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.850796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:24.890233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:24.894648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:24.940329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:24.961201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:24.966531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:25.044038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:25.046914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:25.049824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:25.087105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:25.117437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:25.146426] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:25.166619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F42-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:25.166665] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F42-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:25.166935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F42-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:25.167432] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:25.224103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:25.322775] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:25.355882] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:25.359032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:25.359073] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:25.469051] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:25.469099] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667859, retry interval=1000000, ret=-4179) [2022-08-08 10:57:25.473160] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:25.473195] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667864, retry interval=1000000, ret=-4179) [2022-08-08 10:57:25.519078] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:26.174554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F45-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:26.174919] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:26.175240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=267] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:26.175283] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:26.175301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:26.361360] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:26.365117] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:26.365254] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:26.502850] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:26.502900] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667858, retry interval=1000000, ret=-4179) [2022-08-08 10:57:26.513834] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:26.513889] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667863, retry interval=1000000, ret=-4179) [2022-08-08 10:57:26.823340] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927446819152 2022-08-08 02:57:26.819152|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:10657038013414801684, inc:84093901, addr:{ip:"192.168.11.128", port:2882}, t:1659927446819000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABE66-0-0, start_time:1659927446819048, snapshot:1659927446721695|[end_task] u=115 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927446819283|[calc_checksum_by_commit] u=17 ret:0, checksum:4052630531, checksum_log_ts:9223372036854775807|[submit_log] u=120 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=3 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3800 trans_version:1659927446819027, uref:1073741825|[exiting] u=17 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315639, logging:0, pending:0, uref:1|[end_trans_cb] u=19 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=4116) [2022-08-08 10:57:26.823494] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:26.876464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:27.368949] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:27.372188] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:27.372223] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:27.542287] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927447542283, cur_ts_=1659927447516835, delta=25448) [2022-08-08 10:57:27.561664] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:27.561697] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667857, retry interval=1000000, ret=-4179) [2022-08-08 10:57:27.562303] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:27.562365] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667862, retry interval=1000000, ret=-4179) [2022-08-08 10:57:27.826642] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10235, time_dist: generate_cluster_version=10230) [2022-08-08 10:57:28.015738] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.015914] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.016062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.016331] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.094099] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E6FEB-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.094228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E6FEB-0-0] [lt=55] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.296617] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E70C5-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.296635] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E70C5-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.296640] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E70C5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.296686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E70C5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:28.375674] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:28.400422] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927448400414, cur_ts_=1659927448376623, delta=23791) [2022-08-08 10:57:28.403521] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:28.403557] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:28.483466] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=20558) [2022-08-08 10:57:28.497987] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14486) [2022-08-08 10:57:28.500181] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37799, time_dist: generate_cluster_version=37789, cluster_service_self_check=6) [2022-08-08 10:57:28.501140] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18697, time_dist: do_cluster_heartbeat=10, generate_cluster_version=18684) [2022-08-08 10:57:28.608082] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:28.609105] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667856, retry interval=1000000, ret=-4179) [2022-08-08 10:57:28.634732] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:28.634805] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667861, retry interval=1000000, ret=-4179) [2022-08-08 10:57:28.639518] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30711) [2022-08-08 10:57:28.658459] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18110) [2022-08-08 10:57:28.732560] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927448732552, cur_ts_=1659927448698273, delta=34279) [2022-08-08 10:57:28.884482] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927448884476, cur_ts_=1659927448863455, delta=21021) [2022-08-08 10:57:29.112275] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14854) [2022-08-08 10:57:29.112682] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17043, time_dist: generate_cluster_version=17040) [2022-08-08 10:57:29.122500] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10178) [2022-08-08 10:57:29.122936] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26503, time_dist: generate_cluster_version=26501) [2022-08-08 10:57:29.418748] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:29.423568] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:29.423630] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:29.614546] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:29.614587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:29.633348] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11090) [2022-08-08 10:57:29.634487] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19023, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19011) [2022-08-08 10:57:29.635495] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:29.636088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:29.636492] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:29.636524] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667855, retry interval=1000000, ret=-4179) [2022-08-08 10:57:29.657686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:29.657706] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:29.658869] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:29.658921] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667860, retry interval=1000000, ret=-4179) [2022-08-08 10:57:29.722711] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:57:29.722750] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:57:29.846728] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=18743) [2022-08-08 10:57:29.847558] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=13] [dc=0] [trans sampling] (TRACE=begin_ts=1659927449827707 2022-08-08 02:57:29.827707|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:6078375848707765471, inc:84094875, addr:{ip:"192.168.11.128", port:2882}, t:1659927449826000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=8 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A60A8-0-0, start_time:1659927449827532, snapshot:1659927449795496|[end_task] u=188 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=13 stc:1659927449827918|[calc_checksum_by_commit] u=28 ret:0, checksum:259346417, checksum_log_ts:9223372036854775807|[submit_log] u=18808 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=6 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=681 trans_version:1659927449827948, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301062, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=19764) [2022-08-08 10:57:29.847824] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20628, time_dist: generate_cluster_version=20627) [2022-08-08 10:57:29.971939] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14785088}) [2022-08-08 10:57:30.000285] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927450000278, cur_ts_=1659927449973095, delta=27183) [2022-08-08 10:57:30.184740] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=175] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927448276191, end_time=1659927450184727, elapsed_time=1908536, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:57:30.209850] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:30.209906] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:30.425402] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:30.430017] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:30.430056] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:30.453335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:30.460657] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17611, time_dist: do_cluster_heartbeat=8, generate_cluster_version=17599) [2022-08-08 10:57:30.461848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:30.640650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:30.659697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:30.659720] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:30.661407] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:30.661528] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667854, retry interval=1000000, ret=-4179) [2022-08-08 10:57:30.668266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6107-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:30.668401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6107-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:30.685064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:30.685177] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:30.686353] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:30.686413] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667859, retry interval=1000000, ret=-4179) [2022-08-08 10:57:30.779136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:30.841198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:30.852049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:30.854432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:30.893259] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:30.929543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:30.930450] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:30.940421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:30.965649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:30.986392] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:31.012350] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:31.091360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:31.158009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:31.219889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F54-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.219929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F54-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.220179] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F54-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.220299] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F54-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.220472] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.220615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.220757] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.220870] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:31.324197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:31.326255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:31.376235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:31.432260] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:31.435810] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:31.435847] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:31.689861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4902-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:31.694202] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:31.694241] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667853, retry interval=1000000, ret=-4179) [2022-08-08 10:57:31.704992] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11406) [2022-08-08 10:57:31.707610] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14646, time_dist: generate_cluster_version=14638) [2022-08-08 10:57:31.717005] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:31.717042] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667858, retry interval=1000000, ret=-4179) [2022-08-08 10:57:31.720736] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15596) [2022-08-08 10:57:31.722288] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26494, time_dist: generate_cluster_version=26490) [2022-08-08 10:57:32.144557] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927452144547, cur_ts_=1659927452121292, delta=23255) [2022-08-08 10:57:32.230428] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F57-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:32.230461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F57-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:32.230823] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F57-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:32.232013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:32.438665] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:32.442304] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:32.442354] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:32.485882] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12324, time_dist: do_cluster_heartbeat=11, generate_cluster_version=12308) [2022-08-08 10:57:32.679915] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:32.702065] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:32.702201] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:32.719530] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:32.719550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:32.720777] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:32.720874] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667852, retry interval=1000000, ret=-4179) [2022-08-08 10:57:32.737059] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:32.737162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:32.738548] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:32.738627] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667857, retry interval=1000000, ret=-4179) [2022-08-08 10:57:32.771695] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=931] [dc=0] clock out of order(cur_ts=1659927452771688, cur_ts_=1659927452748690, delta=22998) [2022-08-08 10:57:32.891623] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927452891614, cur_ts_=1659927452851662, delta=39952) [2022-08-08 10:57:32.931232] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=97966) [2022-08-08 10:57:32.931306] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=30] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:15277293366198415074, inc:84095008, addr:"192.168.11.128:2882", t:1659927452805927}, log_id=8301077, used_time=107136) [2022-08-08 10:57:32.931486] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=108398, time_dist: generate_cluster_version=108396) [2022-08-08 10:57:32.948693] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17414) [2022-08-08 10:57:32.948759] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=26] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:17050005550945748220, inc:84095011, addr:"192.168.11.128:2882", t:1659927452805927}, log_id=8315669, used_time=116439) [2022-08-08 10:57:32.955984] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=124881, time_dist: generate_cluster_version=124877) [2022-08-08 10:57:32.991841] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927452991831, cur_ts_=1659927452955138, delta=36693) [2022-08-08 10:57:33.445236] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:33.450696] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:33.450825] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:33.750246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:33.751594] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:33.751654] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667851, retry interval=1000000, ret=-4179) [2022-08-08 10:57:33.762646] WARN [COMMON] on_full_return (ob_io_struct.cpp:2546) [10864][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] push io request into callback queue failed(ret=-4072, req={is_inited_:true, is_finished_:true, is_canceled_:true, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:492}, offset_:1306624, size_:118784, flag_:{mode_:0, category_:1, wait_event_id_:1, reserved_:0}, write_buf_:null, callback_:0x7f6969e220d0}, deadline_ts_:1659927453762233, control_block_:0x7f6833bc4560, raw_buf_:null, io_buf_:0x7f67cf6d7000, io_offset_:1306624, io_size_:118784, complete_size_:118784, time_log_:{begin_ts_:1659927453762231, enqueue_ts_:1659927453762234, dequeue_ts_:1659927453762278, submit_ts_:1659927453762282, return_ts_:1659927453762619, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1659927453762620}, channel_:0x7f69a4ef6310, ref_cnt_:2, out_ref_cnt_:0, trace_id_:Y0-0000000000000000-0-0, ret_code_:{io_ret_:-4072, fs_errno_:0}, retry_count_:0, copied_callback_:0x7f67a322b0d0}) [2022-08-08 10:57:33.762689] WARN [COMMON] get_events (ob_io_struct.cpp:2500) [10864][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] process full return io request failed(ret=-4072, *req={is_inited_:true, is_finished_:true, is_canceled_:true, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:492}, offset_:1306624, size_:118784, flag_:{mode_:0, category_:1, wait_event_id_:1, reserved_:0}, write_buf_:null, callback_:0x7f6969e220d0}, deadline_ts_:1659927453762233, control_block_:0x7f6833bc4560, raw_buf_:null, io_buf_:0x7f67cf6d7000, io_offset_:1306624, io_size_:118784, complete_size_:118784, time_log_:{begin_ts_:1659927453762231, enqueue_ts_:1659927453762234, dequeue_ts_:1659927453762278, submit_ts_:1659927453762282, return_ts_:1659927453762619, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1659927453762620}, channel_:0x7f69a4ef6310, ref_cnt_:1, out_ref_cnt_:0, trace_id_:Y0-0000000000000000-0-0, ret_code_:{io_ret_:-4072, fs_errno_:0}, retry_count_:0, copied_callback_:0x7f67a322b0d0}) [2022-08-08 10:57:33.764213] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:33.764244] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667856, retry interval=1000000, ret=-4179) [2022-08-08 10:57:34.124269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:34.250798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=1916] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.287356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.297774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.424223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.454447] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:34.457699] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:34.457741] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:34.467345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.467582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:34.473270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:34.487951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.497116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:34.500646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.577527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.604575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:34.614613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.624041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.625094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.629550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.646266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:34.656374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:34.669667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.684182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.690112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:34.708109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.709875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.727751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.739249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.740581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.741086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:34.754007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.755257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.760277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.760901] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:34.760938] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:34.762976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.764047] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.773042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.788753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.790313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.791458] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:34.791499] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667855, retry interval=1000000, ret=-4179) [2022-08-08 10:57:34.793803] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:34.795148] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:34.795177] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667850, retry interval=1000000, ret=-4179) [2022-08-08 10:57:34.804096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.830429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:34.851830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.859225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.865279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.875595] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:34.905403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:34.916943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:34.959822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:34.980717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:34.981674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.064517] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.065532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.075023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:35.106784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.132575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:35.169999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.240062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:35.342787] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.460079] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:35.466431] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:35.466474] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:35.543669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:35.736471] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:57:35.794185] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4945-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:35.829893] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927455829881, cur_ts_=1659927455805433, delta=24448) [2022-08-08 10:57:35.847857] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:35.848229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:35.849519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:35.849574] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:35.852819] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:35.852886] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667849, retry interval=1000000, ret=-4179) [2022-08-08 10:57:35.853659] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:35.853685] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667854, retry interval=1000000, ret=-4179) [2022-08-08 10:57:35.854257] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=6] [dc=0] [trans sampling] (TRACE=begin_ts=1659927455853383 2022-08-08 02:57:35.853383|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:925431850870669603, inc:84095119, addr:{ip:"192.168.11.128", port:2882}, t:1659927455853000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7CE2-0-0, start_time:1659927455853268, snapshot:1659927455841144|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927455853525|[calc_checksum_by_commit] u=17 ret:0, checksum:3072225045, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=508 trans_version:1659927455853152, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987819, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=759) [2022-08-08 10:57:36.469239] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:36.472568] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:36.472614] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:36.893255] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:36.917297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:36.917399] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] clock out of order(cur_ts=1659927456917394, cur_ts_=1659927456885850, delta=31544) [2022-08-08 10:57:36.930925] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:36.930978] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667853, retry interval=1000000, ret=-4179) [2022-08-08 10:57:36.931819] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=37884, time_dist: generate_cluster_version=37882) [2022-08-08 10:57:36.932843] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:36.932966] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:36.935364] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:36.935399] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667848, retry interval=1000000, ret=-4179) [2022-08-08 10:57:37.106571] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927457106564, cur_ts_=1659927457072804, delta=33760) [2022-08-08 10:57:37.479430] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:37.483847] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:37.483896] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:37.939126] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] clock out of order(cur_ts=1659927457939120, cur_ts_=1659927457910081, delta=29039) [2022-08-08 10:57:37.951615] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=41585) [2022-08-08 10:57:37.952855] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43552, time_dist: generate_cluster_version=43550) [2022-08-08 10:57:38.090937] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927458090932, cur_ts_=1659927458064600, delta=26332) [2022-08-08 10:57:38.166551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:38.166599] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:38.170258] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:38.170336] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667847, retry interval=1000000, ret=-4179) [2022-08-08 10:57:38.171713] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=26491) [2022-08-08 10:57:38.184325] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:38.184359] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667852, retry interval=1000000, ret=-4179) [2022-08-08 10:57:38.185349] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=40884, time_dist: do_cluster_heartbeat=12, generate_cluster_version=40868) [2022-08-08 10:57:38.187341] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10453, time_dist: generate_cluster_version=10451) [2022-08-08 10:57:38.297475] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927458297466, cur_ts_=1659927458246226, delta=51240) [2022-08-08 10:57:38.385571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F69-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:38.385618] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F69-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:38.385633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F69-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:38.385641] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F69-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:38.385749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:38.486738] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:38.490853] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:38.490877] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:38.703129] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11319) [2022-08-08 10:57:38.703386] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12355, time_dist: do_cluster_heartbeat=11, generate_cluster_version=12341) [2022-08-08 10:57:38.761694] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11247) [2022-08-08 10:57:38.761964] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12215, time_dist: generate_cluster_version=12212) [2022-08-08 10:57:38.852570] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927458852561, cur_ts_=1659927458819938, delta=32623) [2022-08-08 10:57:38.936779] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=6] [dc=0] [trans sampling] (TRACE=begin_ts=1659927458926942 2022-08-08 02:57:38.926942|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:13866823159946683905, inc:84095235, addr:{ip:"192.168.11.128", port:2882}, t:1659927458897000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A60D4-0-0, start_time:1659927458926816, snapshot:1659927458852081|[end_task] u=130 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927458927091|[calc_checksum_by_commit] u=22 ret:0, checksum:2012793528, checksum_log_ts:9223372036854775807|[submit_log] u=31 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=9564 trans_version:1659927458927114, uref:1073741825|[exiting] u=17 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301106, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=9797) [2022-08-08 10:57:38.936913] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10479, time_dist: do_cluster_heartbeat=4, generate_cluster_version=10473) [2022-08-08 10:57:39.210037] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927459210033, cur_ts_=1659927459175487, delta=34546) [2022-08-08 10:57:39.226528] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:39.226584] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667851, retry interval=1000000, ret=-4179) [2022-08-08 10:57:39.231425] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:39.231483] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667846, retry interval=1000000, ret=-4179) [2022-08-08 10:57:39.470970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F6C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.471009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F6C-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.471068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F6C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.473453] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F6C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.473577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.473604] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.473616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:39.502033] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:39.513680] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32101) [2022-08-08 10:57:39.514140] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34903, time_dist: do_cluster_heartbeat=11, generate_cluster_version=34888) [2022-08-08 10:57:39.527458] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:39.527504] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:39.533962] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20236) [2022-08-08 10:57:39.534159] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30472, time_dist: do_cluster_heartbeat=10, generate_cluster_version=30460) [2022-08-08 10:57:39.559585] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927459559575, cur_ts_=1659927459536937, delta=22638) [2022-08-08 10:57:39.569215] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31273) [2022-08-08 10:57:39.606229] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15692) [2022-08-08 10:57:39.698096] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927459698085, cur_ts_=1659927459668312, delta=29773) [2022-08-08 10:57:39.726815] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:57:39.726891] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:57:39.775245] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13370) [2022-08-08 10:57:39.775752] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15187, time_dist: generate_cluster_version=15184) [2022-08-08 10:57:39.808153] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] clock out of order(cur_ts=1659927459808146, cur_ts_=1659927459787335, delta=20811) [2022-08-08 10:57:39.828025] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18507) [2022-08-08 10:57:39.828477] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19828, time_dist: print_stat=7, do_cluster_heartbeat=3, generate_cluster_version=19816) [2022-08-08 10:57:40.284681] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:40.284735] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667850, retry interval=1000000, ret=-4179) [2022-08-08 10:57:40.295945] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:40.295999] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667845, retry interval=1000000, ret=-4179) [2022-08-08 10:57:40.398821] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14789408}) [2022-08-08 10:57:40.471712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:40.492771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:40.543604] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:40.572685] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=61181) [2022-08-08 10:57:40.582390] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927460582384, cur_ts_=1659927460549338, delta=33046) [2022-08-08 10:57:40.591570] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=80776, time_dist: do_cluster_heartbeat=10, generate_cluster_version=80762) [2022-08-08 10:57:40.601479] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28753) [2022-08-08 10:57:40.609345] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=81115, time_dist: do_cluster_heartbeat=10, generate_cluster_version=81103) [2022-08-08 10:57:40.611764] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:40.611800] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:40.663475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:40.704151] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927458323562, end_time=1659927460704138, elapsed_time=2380576, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:57:40.803701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:40.863299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:40.874826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:40.875483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:40.908982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:40.946421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:40.951172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:40.960723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:40.981438] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:41.002715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:41.028655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:41.116041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:41.174419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:41.306074] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.306317] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=110] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.306461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.306600] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.306694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=81] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.306894] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.307269] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.307474] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.307507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6116-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:41.327712] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:41.328448] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] task process failed, start retry(max retry time=9223372036854667844, retry interval=1000000, ret=-4179) [2022-08-08 10:57:41.328546] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:41.328616] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] task process failed, start retry(max retry time=9223372036854667849, retry interval=1000000, ret=-4179) [2022-08-08 10:57:41.347151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:41.347265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:41.395849] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:41.631035] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:41.640908] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:41.640935] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:41.909401] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] [trans sampling] (TRACE=begin_ts=1659927461908549 2022-08-08 02:57:41.908549|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17709617082593353600, inc:84095345, addr:{ip:"192.168.11.128", port:2882}, t:1659927461908000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=25 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A60E3-0-0, start_time:1659927461908433, snapshot:1659927461881567|[end_task] u=127 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927461908714|[calc_checksum_by_commit] u=20 ret:0, checksum:872458343, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=540 trans_version:1659927461908735, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301121, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=814) [2022-08-08 10:57:42.066578] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927462066573, cur_ts_=1659927462039594, delta=26979) [2022-08-08 10:57:42.360645] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:42.360694] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667843, retry interval=1000000, ret=-4179) [2022-08-08 10:57:42.363075] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:42.363137] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667848, retry interval=1000000, ret=-4179) [2022-08-08 10:57:42.643847] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:42.649502] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:42.649547] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:43.340209] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27536) [2022-08-08 10:57:43.342358] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30637, time_dist: generate_cluster_version=30633) [2022-08-08 10:57:43.367004] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6119-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:43.367065] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6119-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:43.392032] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:43.392069] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667847, retry interval=1000000, ret=-4179) [2022-08-08 10:57:43.396140] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:43.396172] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667842, retry interval=1000000, ret=-4179) [2022-08-08 10:57:43.652823] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:43.656312] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:43.656360] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:44.001283] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10994) [2022-08-08 10:57:44.001582] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20103, time_dist: generate_cluster_version=20100) [2022-08-08 10:57:44.144429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:44.272027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.306085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.320606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.419776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:44.419922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:44.421257] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:44.421304] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667846, retry interval=1000000, ret=-4179) [2022-08-08 10:57:44.425434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:44.429096] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:44.429149] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667841, retry interval=1000000, ret=-4179) [2022-08-08 10:57:44.444814] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.483118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:44.485059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.492657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:44.506075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.518364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:44.524463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.602227] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.627558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:44.635901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.641977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=194] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.647998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.652778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.658357] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:44.661469] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:44.661520] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:44.668811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:44.679186] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:44.686497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.702356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.712899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:44.746633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.746649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.746790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.748007] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927464748001, cur_ts_=1659927464718706, delta=29295) [2022-08-08 10:57:44.757207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.758640] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:44.758886] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11125) [2022-08-08 10:57:44.759182] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12134, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12123) [2022-08-08 10:57:44.759240] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12105, time_dist: do_cluster_heartbeat=6, generate_cluster_version=12098) [2022-08-08 10:57:44.759253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.769855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.778836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.779504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.780794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.780892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.789993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.805369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.805730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.826452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.848401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:44.869225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.879458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.889544] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.901922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:44.921264] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:44.923956] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927464922126 2022-08-08 02:57:44.922126|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:16397131082691348929, inc:84095456, addr:{ip:"192.168.11.128", port:2882}, t:1659927464920000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABEBF-0-0, start_time:1659927464921984, snapshot:1659927464916702|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927464922269|[calc_checksum_by_commit] u=17 ret:0, checksum:150060496, checksum_log_ts:9223372036854775807|[submit_log] u=29 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1564 trans_version:1659927464920596, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315728, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=1792) [2022-08-08 10:57:44.936745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:44.976922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:44.997596] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:45.007739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.082347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.082470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.092811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:45.124045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.153761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:45.194984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.258921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:45.362725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.424583] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A96-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:45.424652] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4A96-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:45.451118] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:45.451171] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667840, retry interval=1000000, ret=-4179) [2022-08-08 10:57:45.453953] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:45.454012] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667845, retry interval=1000000, ret=-4179) [2022-08-08 10:57:45.564102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:45.666365] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:45.669789] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:45.669869] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:46.482358] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:46.482389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:46.483262] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:46.483294] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667844, retry interval=1000000, ret=-4179) [2022-08-08 10:57:46.484069] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:46.484095] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667839, retry interval=1000000, ret=-4179) [2022-08-08 10:57:46.672391] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:46.675381] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:46.675417] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:46.941998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:46.993686] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=19708] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:47.509090] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:47.509141] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667838, retry interval=1000000, ret=-4179) [2022-08-08 10:57:47.521235] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:47.521384] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] task process failed, start retry(max retry time=9223372036854667843, retry interval=1000000, ret=-4179) [2022-08-08 10:57:47.682719] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:47.686353] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:47.686412] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:47.953699] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927467952559 2022-08-08 02:57:47.952559|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:2724596130843912267, inc:84095580, addr:{ip:"192.168.11.128", port:2882}, t:1659927467951000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6101-0-0, start_time:1659927467952430, snapshot:1659927467927568|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927467952701|[calc_checksum_by_commit] u=21 ret:0, checksum:979471148, checksum_log_ts:9223372036854775807|[submit_log] u=56 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=845 trans_version:1659927467952722, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301151, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1102) [2022-08-08 10:57:48.404086] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11513, time_dist: generate_cluster_version=11510) [2022-08-08 10:57:48.548537] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:48.548572] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667837, retry interval=1000000, ret=-4179) [2022-08-08 10:57:48.552767] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:48.552989] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667842, retry interval=1000000, ret=-4179) [2022-08-08 10:57:48.689716] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:48.693437] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:48.693483] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:48.708281] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F87-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:48.708391] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F87-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:48.708697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F87-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:48.709607] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F87-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:48.710158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=195] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:48.710192] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:48.710207] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:49.603444] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:49.603481] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667836, retry interval=1000000, ret=-4179) [2022-08-08 10:57:49.611756] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:49.611798] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667841, retry interval=1000000, ret=-4179) [2022-08-08 10:57:49.699091] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:49.703732] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:49.707001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=3246] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:49.724152] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:57:49.724170] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:57:49.911990] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14793664}) [2022-08-08 10:57:49.984545] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19949, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19936) [2022-08-08 10:57:50.012108] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927470012101, cur_ts_=1659927469986888, delta=25213) [2022-08-08 10:57:50.025452] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=41116) [2022-08-08 10:57:50.031686] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=66244, time_dist: generate_cluster_version=66239) [2022-08-08 10:57:50.159390] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927468305195, end_time=1659927470159377, elapsed_time=1854182, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:57:50.488707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=837] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:50.515572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:50.614752] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C495A-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.614788] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C495A-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.614938] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C495A-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.615389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C495A-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.627832] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.627868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.629612] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:50.629645] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667835, retry interval=1000000, ret=-4179) [2022-08-08 10:57:50.639017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.639092] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:50.640578] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:50.640611] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667840, retry interval=1000000, ret=-4179) [2022-08-08 10:57:50.688152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:50.709443] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:50.712650] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:50.712707] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:50.827054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:50.887715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:50.897987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:50.899984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:50.949121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:50.951099] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927470951093, cur_ts_=1659927470926676, delta=24417) [2022-08-08 10:57:50.961699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:50.974139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:50.982727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=154] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:51.002824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:51.004683] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=122] [dc=0] [trans sampling] (TRACE=begin_ts=1659927471003533 2022-08-08 02:57:51.003533|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:10199680338019152034, inc:84095690, addr:{ip:"192.168.11.128", port:2882}, t:1659927471002000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6110-0-0, start_time:1659927471003389, snapshot:1659927470958933|[end_task] u=158 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927471003712|[calc_checksum_by_commit] u=25 ret:0, checksum:2322192670, checksum_log_ts:9223372036854775807|[submit_log] u=68 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=806 trans_version:1659927471003739, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301166, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1107) [2022-08-08 10:57:51.025120] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:51.045754] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:51.138453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:51.194665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:51.365709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:51.369123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:51.417744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:51.663400] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:51.663446] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667839, retry interval=1000000, ret=-4179) [2022-08-08 10:57:51.668519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:51.668575] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:51.670031] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:51.670066] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667834, retry interval=1000000, ret=-4179) [2022-08-08 10:57:51.716165] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:51.719930] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:51.719970] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:52.682501] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:52.722294] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:52.729296] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:52.729345] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667833, retry interval=1000000, ret=-4179) [2022-08-08 10:57:52.734256] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:52.734307] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667838, retry interval=1000000, ret=-4179) [2022-08-08 10:57:52.734877] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:52.734893] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:53.739003] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:53.745573] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=774] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11572) [2022-08-08 10:57:53.747269] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:53.747296] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:53.749825] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927473736720 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927473748902 abs_expired_time=1659927475678854 stmt_timeout=1929952 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21592310 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927473728627 start_version=-1 trans_version=1659927473736710 commit_version=0 stmt_start_time=1659927473736710 abs_expired_time=1659927503647698 stmt_timeout=29910988 abs_lock_wait_timeout=1659927503647698 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:13255479632449870218, inc:84095792, addr:"192.168.11.128:2882", t:1659927473736710} checksum_log_ts=9223372036854775807, version=1659927473736720) [2022-08-08 10:57:53.768139] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:53.768196] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667832, retry interval=1000000, ret=-4179) [2022-08-08 10:57:53.769479] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:53.769509] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667837, retry interval=1000000, ret=-4179) [2022-08-08 10:57:54.017544] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=36] [dc=0] [trans sampling] (TRACE=begin_ts=1659927474016776 2022-08-08 02:57:54.016776|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:204444194190452156, inc:84095804, addr:{ip:"192.168.11.128", port:2882}, t:1659927474015000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABEEC-0-0, start_time:1659927474016626, snapshot:1659927473969728|[end_task] u=123 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927474016916|[calc_checksum_by_commit] u=17 ret:0, checksum:2598551083, checksum_log_ts:9223372036854775807|[submit_log] u=43 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=495 trans_version:1659927474016761, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=1 ret:0, log_type:64, log_id:8315773, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=729) [2022-08-08 10:57:54.160330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:54.289580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.328315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.345111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.468106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.505931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:54.507359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.514272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:54.528507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.534693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:54.549514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.627309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.649755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:54.653639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.664047] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.670581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.670702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.691383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:54.699918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:54.701616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=162] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.718312] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.734232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:54.751329] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:54.755420] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:54.755900] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=464] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:54.762322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.763982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.764035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.776454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.776916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4925-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:54.777385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.778889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:57:54.792046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.800361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.801144] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:54.801194] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667831, retry interval=1000000, ret=-4179) [2022-08-08 10:57:54.801357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.804211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.804275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.808774] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:54.808844] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667836, retry interval=1000000, ret=-4179) [2022-08-08 10:57:54.811869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.824241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=731] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.824271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=851] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.845668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.865807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=6867] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:54.900750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.902707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:54.912130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.921333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:57:54.942128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:54.952401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:54.995017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:57:55.020225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:55.033129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.098841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.100497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.114555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:55.145216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.176471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:57:55.213146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.282731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:57:55.377868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.582707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:57:55.762966] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:55.769945] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:55.769985] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:55.774027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F9C-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:55.779074] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=217] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:57:55.805634] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927475679971 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927475764042 abs_expired_time=1659927477693498 stmt_timeout=1929456 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21592335 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927475679961 start_version=-1 trans_version=0 commit_version=0 stmt_start_time=1659927475752257 abs_expired_time=1659927505654098 stmt_timeout=29901841 abs_lock_wait_timeout=1659927505654098 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:14378231768774493972, inc:84095857, addr:"192.168.11.128:2882", t:1659927475752257} checksum_log_ts=9223372036854775807, version=1659927475679971) [2022-08-08 10:57:55.805870] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.805900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.805962] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806146] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806169] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806890] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.806898] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC612B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:55.837916] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:55.837952] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667835, retry interval=1000000, ret=-4179) [2022-08-08 10:57:55.839853] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11772, time_dist: generate_cluster_version=11761, cluster_service_self_check=6) [2022-08-08 10:57:55.844658] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:55.844691] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667830, retry interval=1000000, ret=-4179) [2022-08-08 10:57:55.844797] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=218] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17371, time_dist: generate_cluster_version=17368) [2022-08-08 10:57:56.774063] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:56.777519] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:56.777555] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:56.819390] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F9F-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:56.819418] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F9F-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:56.821315] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2F9F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:56.868042] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=422] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:56.868206] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667834, retry interval=1000000, ret=-4179) [2022-08-08 10:57:56.873423] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:56.873472] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667829, retry interval=1000000, ret=-4179) [2022-08-08 10:57:56.960750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:57:57.047191] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:57.048712] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=21] [dc=0] [trans sampling] (TRACE=begin_ts=1659927477047723 2022-08-08 02:57:57.047723|[init] u=0 arg1:140085232810448, ctx_type:"participant", trans_type:0, trans_id:{hash:8889497788232043273, inc:84095911, addr:{ip:"192.168.11.128", port:2882}, t:1659927477046000}, pkey:{tid:1099511628085, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC40F63DD-0-0, start_time:1659927477047614, snapshot:1659927476993976|[end_task] u=255 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927477047996|[calc_checksum_by_commit] u=17 ret:0, checksum:4173361579, checksum_log_ts:9223372036854775807|[submit_log] u=46 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=578 trans_version:1659927477046853, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:1160586, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232811824|total_timeu=949) [2022-08-08 10:57:57.779952] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:57.783313] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:57.783420] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:57.827168] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2FA2-0-0] [lt=71] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927477825595 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927477826813 abs_expired_time=1659927479756479 stmt_timeout=1929666 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21592371 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927477816879 start_version=-1 trans_version=1659927477825585 commit_version=0 stmt_start_time=1659927477818169 abs_expired_time=1659927507719430 stmt_timeout=29901261 abs_lock_wait_timeout=1659927507719430 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:7538310196725708086, inc:84095946, addr:"192.168.11.128:2882", t:1659927477818169} checksum_log_ts=9223372036854775807, version=1659927477825595) [2022-08-08 10:57:57.870377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AA8-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:57.870977] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AA8-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:57.870993] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AA8-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:57.871146] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AA8-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:57:57.904638] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:57.904705] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667828, retry interval=1000000, ret=-4179) [2022-08-08 10:57:57.905335] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:57.905359] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667833, retry interval=1000000, ret=-4179) [2022-08-08 10:57:58.233867] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23290, time_dist: generate_cluster_version=23286) [2022-08-08 10:57:58.785898] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:58.791951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:58.791972] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:58.843393] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14057, time_dist: generate_cluster_version=14052) [2022-08-08 10:57:58.920651] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927478920646, cur_ts_=1659927478877564, delta=43082) [2022-08-08 10:57:58.998185] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:58.998240] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667832, retry interval=1000000, ret=-4179) [2022-08-08 10:57:59.002669] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:57:59.002728] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667827, retry interval=1000000, ret=-4179) [2022-08-08 10:57:59.187318] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927479187309, cur_ts_=1659927479158304, delta=29005) [2022-08-08 10:57:59.723947] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:57:59.724001] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:57:59.793704] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:57:59.796886] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:57:59.796943] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:57:59.847401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2FA8-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:59.847500] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:59.847512] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:59.847576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:57:59.860256] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10857) [2022-08-08 10:57:59.893200] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14798048}) [2022-08-08 10:58:00.046020] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14175, time_dist: print_stat=8, generate_cluster_version=14162) [2022-08-08 10:58:00.058564] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:00.058598] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667831, retry interval=1000000, ret=-4179) [2022-08-08 10:58:00.075114] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:00.075148] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667826, retry interval=1000000, ret=-4179) [2022-08-08 10:58:00.075497] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15016) [2022-08-08 10:58:00.075678] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=48] [dc=0] [trans sampling] (TRACE=begin_ts=1659927480060227 2022-08-08 02:58:00.060227|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:6653885269691270896, inc:84096031, addr:{ip:"192.168.11.128", port:2882}, t:1659927480052000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7D10-0-0, start_time:1659927480060114, snapshot:1659927480043727|[end_task] u=161 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927480060404|[calc_checksum_by_commit] u=17 ret:0, checksum:1565286749, checksum_log_ts:9223372036854775807|[submit_log] u=29 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=15141 trans_version:1659927480052806, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987865, logging:0, pending:0, uref:1|[end_trans_cb] u=24 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=15414) [2022-08-08 10:58:00.085667] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10119) [2022-08-08 10:58:00.086230] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16045, time_dist: do_cluster_heartbeat=7, generate_cluster_version=16035) [2022-08-08 10:58:00.139252] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927478306697, end_time=1659927480139238, elapsed_time=1832541, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:58:00.521207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:00.521384] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927480521377, cur_ts_=1659927480496320, delta=25057) [2022-08-08 10:58:00.532246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:00.710415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:00.798707] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:00.803494] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:00.803632] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:00.850302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:00.902933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:00.913269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:00.924134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:00.965365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:01.009094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:01.009124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:01.009375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:01.009531] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927481009528, cur_ts_=1659927480973286, delta=36242) [2022-08-08 10:58:01.019488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:01.041679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:01.065371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:01.076268] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10148, time_dist: generate_cluster_version=10146) [2022-08-08 10:58:01.077149] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4969-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:01.077277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4969-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:01.078632] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:01.078822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:01.080635] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:01.080692] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667830, retry interval=1000000, ret=-4179) [2022-08-08 10:58:01.096648] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:01.096951] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15395) [2022-08-08 10:58:01.098457] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:01.098511] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667825, retry interval=1000000, ret=-4179) [2022-08-08 10:58:01.160289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:01.212368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:01.386437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:01.386436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:01.440044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:01.471381] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:58:01.622491] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] clock out of order(cur_ts=1659927481622484, cur_ts_=1659927481601958, delta=20526) [2022-08-08 10:58:01.806151] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:01.809446] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:01.809490] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:02.083107] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AAE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:02.103486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:02.106346] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:02.106400] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667829, retry interval=1000000, ret=-4179) [2022-08-08 10:58:02.122250] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:02.123733] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:02.123792] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667824, retry interval=1000000, ret=-4179) [2022-08-08 10:58:02.812047] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:02.815246] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:02.815283] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:03.129046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:03.129209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=147] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:03.134715] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:03.134858] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667828, retry interval=1000000, ret=-4179) [2022-08-08 10:58:03.147941] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:03.148198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:03.149863] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:03.149938] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667823, retry interval=1000000, ret=-4179) [2022-08-08 10:58:03.259300] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927483259291, cur_ts_=1659927483237605, delta=21686) [2022-08-08 10:58:03.817837] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:03.822331] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:03.822374] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:03.879723] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE2FB4-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:04.153661] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:04.153778] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667827, retry interval=1000000, ret=-4179) [2022-08-08 10:58:04.171821] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:04.171855] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667822, retry interval=1000000, ret=-4179) [2022-08-08 10:58:04.179116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:04.311043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:04.347702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.366122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.490133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.521552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:04.528341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.542278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:04.554053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.555774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:04.574238] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.649949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.670293] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:04.678226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.685527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=557] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:04.691229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.695733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.713492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:04.724960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:04.725277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:04.733751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.757899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:04.781886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.782965] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.784381] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.793504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.799797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:04.801003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:04.809176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.825001] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:04.825340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.826058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.828460] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:04.828480] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:04.828752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.828792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:04.836454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.842424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.842576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.862788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.883195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:04.923336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:04.927898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:04.933131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.940213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:04.960966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:04.974400] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.014490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:05.041802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:05.061881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.118057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.125663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.138601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:05.163615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.174055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C496F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:05.174198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C496F-0-0] [lt=121] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:05.180946] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:05.182470] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:05.182503] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667826, retry interval=1000000, ret=-4179) [2022-08-08 10:58:05.196713] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:05.197374] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:05.197413] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667821, retry interval=1000000, ret=-4179) [2022-08-08 10:58:05.229822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.304191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:05.383515] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927485383441, cur_ts_=1659927485348997, delta=34444) [2022-08-08 10:58:05.401433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.599382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:05.782688] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2950] [dc=0] clock out of order(cur_ts=1659927485782680, cur_ts_=1659927485761536, delta=21144) [2022-08-08 10:58:05.831767] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:05.843485] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:05.843511] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:05.856449] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15487) [2022-08-08 10:58:05.861540] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21805, time_dist: do_cluster_heartbeat=8, generate_cluster_version=21795) [2022-08-08 10:58:06.128780] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] clock out of order(cur_ts=1659927486128771, cur_ts_=1659927486028837, delta=99934) [2022-08-08 10:58:06.240485] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927486240477, cur_ts_=1659927486028837, delta=211640) [2022-08-08 10:58:06.340315] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:06.343530] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=663] [dc=0] task process failed, start retry(max retry time=9223372036854667820, retry interval=1000000, ret=-4179) [2022-08-08 10:58:06.344503] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927486342640 2022-08-08 02:58:06.342640|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:861398279911711216, inc:84096258, addr:{ip:"192.168.11.128", port:2882}, t:1659927486341000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7D1C-0-0, start_time:1659927486342520, snapshot:1659927486337826|[end_task] u=167 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=7 stc:1659927486342823|[calc_checksum_by_commit] u=18 ret:0, checksum:3311927216, checksum_log_ts:9223372036854775807|[submit_log] u=31 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1566 trans_version:1659927486341367, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:987877, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233102640|total_timeu=1823) [2022-08-08 10:58:06.344552] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:06.344589] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667825, retry interval=1000000, ret=-4179) [2022-08-08 10:58:06.856651] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:06.859765] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:06.859807] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:06.980832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:07.112367] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:07.353498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4937-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:07.372205] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:07.372252] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667824, retry interval=1000000, ret=-4179) [2022-08-08 10:58:07.374731] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:07.376238] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:07.376296] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667819, retry interval=1000000, ret=-4179) [2022-08-08 10:58:07.863097] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:07.867768] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:07.867864] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:08.386890] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927488386883, cur_ts_=1659927488358934, delta=27949) [2022-08-08 10:58:08.516277] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927488516266, cur_ts_=1659927488481387, delta=34879) [2022-08-08 10:58:08.559791] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=247795) [2022-08-08 10:58:08.559837] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] slow flush(flush_time=18, io_time=247830, file_id=237, flush_start_offset=50950050, file_offset=50950671) [2022-08-08 10:58:08.560099] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=114] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:6893091260924552203, inc:84096340, addr:"192.168.11.128:2882", t:1659927488303282}, log_id=8301251, used_time=251829) [2022-08-08 10:58:08.560243] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:08.560278] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667823, retry interval=1000000, ret=-4179) [2022-08-08 10:58:08.573638] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=270957, time_dist: generate_cluster_version=270952) [2022-08-08 10:58:08.579969] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20020) [2022-08-08 10:58:08.584465] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:10703699917663074020, inc:84096341, addr:"192.168.11.128:2882", t:1659927488315702}, log_id=8315843, used_time=267790) [2022-08-08 10:58:08.613305] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33291) [2022-08-08 10:58:08.613710] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=297720, time_dist: generate_cluster_version=297717) [2022-08-08 10:58:08.620792] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:08.620994] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667818, retry interval=1000000, ret=-4179) [2022-08-08 10:58:08.870920] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:08.874534] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:08.874558] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:08.948979] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] clock out of order(cur_ts=1659927488948974, cur_ts_=1659927488910586, delta=38388) [2022-08-08 10:58:08.968409] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17029) [2022-08-08 10:58:08.971448] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22760, time_dist: do_cluster_heartbeat=10, generate_cluster_version=22745) [2022-08-08 10:58:08.979920] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11462) [2022-08-08 10:58:08.985224] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21072, time_dist: do_cluster_heartbeat=8, generate_cluster_version=21059) [2022-08-08 10:58:09.356235] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927489355294 2022-08-08 02:58:09.355294|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:6166295674941414827, inc:84096375, addr:{ip:"192.168.11.128", port:2882}, t:1659927489355000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABF37-0-0, start_time:1659927489355180, snapshot:1659927489355118|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=36 stc:1659927489355462|[calc_checksum_by_commit] u=19 ret:0, checksum:2050853212, checksum_log_ts:9223372036854775807|[submit_log] u=27 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=651 trans_version:1659927489355202, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315848, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=901) [2022-08-08 10:58:09.424219] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927489424214, cur_ts_=1659927489392209, delta=32005) [2022-08-08 10:58:09.562707] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4975-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:09.583409] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:09.583460] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667822, retry interval=1000000, ret=-4179) [2022-08-08 10:58:09.640960] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:09.641104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] task process failed, start retry(max retry time=9223372036854667817, retry interval=1000000, ret=-4179) [2022-08-08 10:58:09.724084] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:58:09.724102] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:58:09.876795] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:09.882347] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:09.882379] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:09.908638] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927489908614, cur_ts_=1659927489884436, delta=24178) [2022-08-08 10:58:10.011330] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE2FC6-0-0] [lt=13] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927490005311 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927489948440 abs_expired_time=1659927491908653 stmt_timeout=1960213 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21592535 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927489856992 start_version=-1 trans_version=1659927489948440 commit_version=0 stmt_start_time=1659927489948440 abs_expired_time=1659927519900557 stmt_timeout=29952117 abs_lock_wait_timeout=1659927519900557 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:1390854517170460348, inc:84096398, addr:"192.168.11.128:2882", t:1659927489948440} checksum_log_ts=9223372036854775807, version=1659927490005311) [2022-08-08 10:58:10.011447] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33618, time_dist: do_cluster_heartbeat=33605, generate_cluster_version=11) [2022-08-08 10:58:10.018859] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927490018854, cur_ts_=1659927489948440, delta=70414) [2022-08-08 10:58:10.115648] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=110961) [2022-08-08 10:58:10.115683] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] slow flush(flush_time=12, io_time=110991, file_id=237, flush_start_offset=50963326, file_offset=50965423) [2022-08-08 10:58:10.122693] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:1390854517170460348, inc:84096398, addr:"192.168.11.128:2882", t:1659927489948440}, log_id=1655702, used_time=118113) [2022-08-08 10:58:10.162059] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927490162050, cur_ts_=1659927490064952, delta=97098) [2022-08-08 10:58:10.170340] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=54633) [2022-08-08 10:58:10.170540] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=10] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:3412557821482736437, inc:84096402, addr:"192.168.11.128:2882", t:1659927490061390}, log_id=8301260, used_time=105906) [2022-08-08 10:58:10.171032] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=95737, time_dist: do_cluster_heartbeat=71, generate_cluster_version=95663) [2022-08-08 10:58:10.171098] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=107513, time_dist: do_cluster_heartbeat=12, generate_cluster_version=107501) [2022-08-08 10:58:10.335909] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927490335903, cur_ts_=1659927490311187, delta=24716) [2022-08-08 10:58:10.392617] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14802240}) [2022-08-08 10:58:10.541274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:10.553707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:10.587744] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19922, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19911) [2022-08-08 10:58:10.602940] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927488316743, end_time=1659927490602924, elapsed_time=2286181, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:58:10.612341] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:10.613746] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:10.613814] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667821, retry interval=1000000, ret=-4179) [2022-08-08 10:58:10.643441] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6140-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:10.662190] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:10.662257] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667816, retry interval=1000000, ret=-4179) [2022-08-08 10:58:10.733677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:10.866800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:10.920758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:10.931011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:10.941412] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:10.944834] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:10.944860] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:10.949721] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:10.981863] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:11.019742] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] clock out of order(cur_ts=1659927491019728, cur_ts_=1659927490990288, delta=29440) [2022-08-08 10:58:11.024287] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:11.032512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:11.034668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:11.046652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:11.065738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:11.086542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:11.180423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:11.218667] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16869, time_dist: generate_cluster_version=16865) [2022-08-08 10:58:11.227645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:11.403495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:11.409929] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:11.456122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:11.637245] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:11.637281] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667820, retry interval=1000000, ret=-4179) [2022-08-08 10:58:11.682995] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:11.683081] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667815, retry interval=1000000, ret=-4179) [2022-08-08 10:58:11.949137] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:11.952351] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:11.952389] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:12.420721] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927492420040, cur_ts_=1659927492382272, delta=37768) [2022-08-08 10:58:12.657085] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:12.657161] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667819, retry interval=1000000, ret=-4179) [2022-08-08 10:58:12.682792] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:12.705293] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:12.705355] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667814, retry interval=1000000, ret=-4179) [2022-08-08 10:58:12.954567] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:12.957629] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:12.957672] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:13.280999] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] clock out of order(cur_ts=1659927493280993, cur_ts_=1659927493183843, delta=97150) [2022-08-08 10:58:13.296000] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=126821) [2022-08-08 10:58:13.296046] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=25, io_time=126844, file_id=237, flush_start_offset=50994212, file_offset=50994833) [2022-08-08 10:58:13.296086] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:7176935908189334569, inc:84096523, addr:"192.168.11.128:2882", t:1659927493168072}, log_id=8301277, used_time=126992) [2022-08-08 10:58:13.296350] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=130172, time_dist: generate_cluster_version=130170) [2022-08-08 10:58:13.677093] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:13.677178] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667818, retry interval=1000000, ret=-4179) [2022-08-08 10:58:13.728075] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:13.728142] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667813, retry interval=1000000, ret=-4179) [2022-08-08 10:58:13.960619] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:13.963745] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:13.963864] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:14.163929] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927494163923, cur_ts_=1659927494137234, delta=26689) [2022-08-08 10:58:14.205816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:14.330325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:14.364262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.382115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.511606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.543174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:14.552889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.565032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:14.572245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=1940] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.572290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=289] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:14.598011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.667048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.687942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:14.702184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.702450] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:14.702515] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667817, retry interval=1000000, ret=-4179) [2022-08-08 10:58:14.708957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.710074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:14.719561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=662] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.730188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:14.741956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:14.749724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:14.753091] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:14.753141] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667812, retry interval=1000000, ret=-4179) [2022-08-08 10:58:14.755204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.804673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.804716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.804732] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927494804730, cur_ts_=1659927494769602, delta=35128) [2022-08-08 10:58:14.805286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.823861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=3939] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:14.824371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.824532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:14.859340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.859348] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:14.859412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.859567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:14.859627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.863636] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.888254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.889191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.889372] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.900379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.907644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:14.908171] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16141) [2022-08-08 10:58:14.910376] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19984, time_dist: do_cluster_heartbeat=8, generate_cluster_version=19970) [2022-08-08 10:58:14.946486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:14.949456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:14.957230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.957267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:14.967885] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:14.974671] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:14.974699] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:15.000546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:15.000914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.001588] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] clock out of order(cur_ts=1659927495001583, cur_ts_=1659927494973931, delta=27652) [2022-08-08 10:58:15.033127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:15.065037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:15.078371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.135667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.147895] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.159128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:15.188702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.220672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:15.251907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.327174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:15.419909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.621601] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927495621596, cur_ts_=1659927495561671, delta=59925) [2022-08-08 10:58:15.638525] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:15.708333] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14511, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14499) [2022-08-08 10:58:15.752376] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:15.752414] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667816, retry interval=1000000, ret=-4179) [2022-08-08 10:58:15.779685] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:15.779743] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667811, retry interval=1000000, ret=-4179) [2022-08-08 10:58:15.802491] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:58:15.822102] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=16614) [2022-08-08 10:58:15.834162] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29479, time_dist: do_cluster_heartbeat=8, generate_cluster_version=29469) [2022-08-08 10:58:16.002727] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:16.008516] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:16.008558] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:16.230923] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11411, time_dist: generate_cluster_version=11407) [2022-08-08 10:58:16.311391] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927496311385, cur_ts_=1659927496288432, delta=22953) [2022-08-08 10:58:16.471351] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11297, time_dist: generate_cluster_version=11293) [2022-08-08 10:58:16.774648] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:16.775904] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667815, retry interval=1000000, ret=-4179) [2022-08-08 10:58:16.798907] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:16.798967] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667810, retry interval=1000000, ret=-4179) [2022-08-08 10:58:16.997886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:17.011699] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:17.014882] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:17.014917] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:17.050994] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927497050989, cur_ts_=1659927497029603, delta=21386) [2022-08-08 10:58:17.087973] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26742, time_dist: generate_cluster_version=26738) [2022-08-08 10:58:17.185436] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:17.357028] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=583] [dc=0] clock out of order(cur_ts=1659927497357020, cur_ts_=1659927497311867, delta=45153) [2022-08-08 10:58:17.795290] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:17.795339] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667814, retry interval=1000000, ret=-4179) [2022-08-08 10:58:17.818457] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:17.818494] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667809, retry interval=1000000, ret=-4179) [2022-08-08 10:58:18.017684] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:18.023494] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:18.023556] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:18.094531] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927498094522, cur_ts_=1659927498061296, delta=33226) [2022-08-08 10:58:18.377331] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927498377325, cur_ts_=1659927498355996, delta=21329) [2022-08-08 10:58:18.388156] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32225) [2022-08-08 10:58:18.436501] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=40] [dc=0] [trans sampling] (TRACE=begin_ts=1659927498435489 2022-08-08 02:58:18.435489|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:7735323487971939670, inc:84096725, addr:{ip:"192.168.11.128", port:2882}, t:1659927498434000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6199-0-0, start_time:1659927498435341, snapshot:1659927498393641|[end_task] u=123 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927498435630|[calc_checksum_by_commit] u=21 ret:0, checksum:2947722208, checksum_log_ts:9223372036854775807|[submit_log] u=103 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=618 trans_version:1659927498435652, uref:1073741825|[exiting] u=49 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=5 ret:0, log_type:64, log_id:8301303, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=6 arg1:140085232375600|total_timeu=954) [2022-08-08 10:58:18.670748] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18434) [2022-08-08 10:58:18.671006] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19495, time_dist: generate_cluster_version=19494) [2022-08-08 10:58:18.699155] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17898) [2022-08-08 10:58:18.699353] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19025, time_dist: generate_cluster_version=19022) [2022-08-08 10:58:18.815841] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:18.815894] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667813, retry interval=1000000, ret=-4179) [2022-08-08 10:58:18.841769] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:18.841813] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667808, retry interval=1000000, ret=-4179) [2022-08-08 10:58:18.951699] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927498951690, cur_ts_=1659927498929829, delta=21861) [2022-08-08 10:58:19.026448] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:19.048308] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:19.048348] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:19.565453] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927499565448, cur_ts_=1659927499545365, delta=20083) [2022-08-08 10:58:19.724255] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:58:19.724351] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:58:19.832154] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:19.832193] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667812, retry interval=1000000, ret=-4179) [2022-08-08 10:58:19.859715] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:19.859749] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667807, retry interval=1000000, ret=-4179) [2022-08-08 10:58:19.953247] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14806624}) [2022-08-08 10:58:20.049730] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:20.054028] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:20.054100] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:20.098942] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11533) [2022-08-08 10:58:20.102881] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16885, time_dist: generate_cluster_version=16879) [2022-08-08 10:58:20.134770] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19458) [2022-08-08 10:58:20.135268] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20848, time_dist: generate_cluster_version=20845) [2022-08-08 10:58:20.152137] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927498311261, end_time=1659927500152123, elapsed_time=1840862, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:58:20.559660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:20.569057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:20.757537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:20.853419] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:20.853473] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667811, retry interval=1000000, ret=-4179) [2022-08-08 10:58:20.879744] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:20.879834] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667806, retry interval=1000000, ret=-4179) [2022-08-08 10:58:20.883679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:20.945251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:20.957139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:20.965696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:20.997967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:21.040941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:21.050702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:21.051180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:21.056630] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:21.059673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:21.059713] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:21.062062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:21.085891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:21.106706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:21.196534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:21.249646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:21.427163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:21.430437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:21.483742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:21.494434] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=47] [dc=0] [trans sampling] (TRACE=begin_ts=1659927501483444 2022-08-08 02:58:21.483444|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17229244224670831632, inc:84096833, addr:{ip:"192.168.11.128", port:2882}, t:1659927501477000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A61A8-0-0, start_time:1659927501483302, snapshot:1659927501429609|[end_task] u=171 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927501483635|[calc_checksum_by_commit] u=24 ret:0, checksum:3544148826, checksum_log_ts:9223372036854775807|[submit_log] u=45 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=7339 trans_version:1659927501483661, uref:1073741825|[exiting] u=29 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8301318, logging:0, pending:0, uref:1|[end_trans_cb] u=1369 ret:0, param:0, param:true|[destroy] u=1945 arg1:140085232375600|total_timeu=10950) [2022-08-08 10:58:21.645769] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927501645765, cur_ts_=1659927501581844, delta=63921) [2022-08-08 10:58:21.746569] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2487] [dc=0] clock out of order(cur_ts=1659927501746461, cur_ts_=1659927501721016, delta=25445) [2022-08-08 10:58:21.773939] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=52837) [2022-08-08 10:58:21.811152] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=90967, time_dist: generate_cluster_version=90962) [2022-08-08 10:58:21.844778] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=70765) [2022-08-08 10:58:21.844997] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=100574, time_dist: generate_cluster_version=100572) [2022-08-08 10:58:21.924984] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927501924978, cur_ts_=1659927501904519, delta=20459) [2022-08-08 10:58:21.982478] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:21.982516] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667805, retry interval=1000000, ret=-4179) [2022-08-08 10:58:21.987963] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=82764) [2022-08-08 10:58:21.995427] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:13954883276589779514, inc:84096845, addr:"192.168.11.128:2882", t:1659927501867396}, log_id=8301320, used_time=125164) [2022-08-08 10:58:22.012813] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=145361, time_dist: do_cluster_heartbeat=90, generate_cluster_version=145266) [2022-08-08 10:58:22.043603] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=55582) [2022-08-08 10:58:22.047481] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927502047473, cur_ts_=1659927502020589, delta=26884) [2022-08-08 10:58:22.056896] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=98968, time_dist: do_cluster_heartbeat=9, generate_cluster_version=98954) [2022-08-08 10:58:22.062206] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:22.073639] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:22.073708] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667810, retry interval=1000000, ret=-4179) [2022-08-08 10:58:22.088668] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:22.088700] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:22.096086] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13115) [2022-08-08 10:58:22.113985] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34088, time_dist: do_cluster_heartbeat=7, generate_cluster_version=34074) [2022-08-08 10:58:22.145066] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20041) [2022-08-08 10:58:22.148424] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25244, time_dist: do_cluster_heartbeat=10, generate_cluster_version=25230) [2022-08-08 10:58:22.211486] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11399) [2022-08-08 10:58:22.212315] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12946, time_dist: generate_cluster_version=12942) [2022-08-08 10:58:22.288098] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12767) [2022-08-08 10:58:22.288455] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22426, time_dist: generate_cluster_version=22423) [2022-08-08 10:58:22.409934] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927502409929, cur_ts_=1659927502386994, delta=22935) [2022-08-08 10:58:22.510139] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11873) [2022-08-08 10:58:22.511976] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23642, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23628) [2022-08-08 10:58:22.626129] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927502626119, cur_ts_=1659927502597947, delta=28172) [2022-08-08 10:58:22.718539] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13492) [2022-08-08 10:58:22.721825] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32686, time_dist: do_cluster_heartbeat=10, generate_cluster_version=32661) [2022-08-08 10:58:22.749693] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31087) [2022-08-08 10:58:22.750227] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=60136, time_dist: generate_cluster_version=60134) [2022-08-08 10:58:22.796362] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927502796355, cur_ts_=1659927502771266, delta=25089) [2022-08-08 10:58:23.085926] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927503085921, cur_ts_=1659927503046641, delta=39280) [2022-08-08 10:58:23.116539] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:23.144130] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:23.144169] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667804, retry interval=1000000, ret=-4179) [2022-08-08 10:58:23.151143] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=110595) [2022-08-08 10:58:23.151244] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] slow flush(flush_time=19, io_time=110683, file_id=237, flush_start_offset=51086272, file_offset=51086889) [2022-08-08 10:58:23.151353] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=21] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:18436263513416734592, inc:84096893, addr:"192.168.11.128:2882", t:1659927503038787}, log_id=8315917, used_time=111269) [2022-08-08 10:58:23.159550] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=121342, time_dist: do_cluster_heartbeat=16, generate_cluster_version=121321) [2022-08-08 10:58:23.163595] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=79901, time_dist: do_cluster_heartbeat=10, generate_cluster_version=79887) [2022-08-08 10:58:23.180862] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=481] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:23.180897] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:23.211472] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:23.211516] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667809, retry interval=1000000, ret=-4179) [2022-08-08 10:58:24.162630] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:24.162688] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667803, retry interval=1000000, ret=-4179) [2022-08-08 10:58:24.194295] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:24.200271] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:24.200309] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:24.228470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:24.240673] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:24.240816] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667808, retry interval=1000000, ret=-4179) [2022-08-08 10:58:24.330714] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] clock out of order(cur_ts=1659927504330709, cur_ts_=1659927504299977, delta=30732) [2022-08-08 10:58:24.352203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:24.382044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.402679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.497435] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=8] [dc=0] [trans sampling] (TRACE=begin_ts=1659927504496579 2022-08-08 02:58:24.496579|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:12143616898858912008, inc:84096947, addr:{ip:"192.168.11.128", port:2882}, t:1659927504495000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABF83-0-0, start_time:1659927504496472, snapshot:1659927504444624|[end_task] u=118 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927504496714|[calc_checksum_by_commit] u=19 ret:0, checksum:536947778, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=574 trans_version:1659927504495902, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8315924, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=818) [2022-08-08 10:58:24.528619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.559556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:24.571671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.585782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:24.588201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.595207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:24.620981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:24.691200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.712171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:24.721360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.732816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:24.732827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=164] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.741833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:24.753645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:24.766357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:24.766442] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:24.776877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.822572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.824344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.826460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:24.842296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.843774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:24.845201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:24.877917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:24.878178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:24.878221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.883900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:24.884968] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.885134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.905776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=1729] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.905827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=1386] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:24.909270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=1551] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.915525] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.928148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:24.969806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:24.971869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:24.980199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:24.981168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:25.022826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:25.025320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.054399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:25.083763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:25.098226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.158329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.168651] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.181094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:25.183732] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:25.183798] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667802, retry interval=1000000, ret=-4179) [2022-08-08 10:58:25.202788] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:25.205908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.206095] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:25.206128] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:25.244283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:25.262035] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:25.262098] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667807, retry interval=1000000, ret=-4179) [2022-08-08 10:58:25.274909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.342650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:25.442533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.660467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:25.890079] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10536, time_dist: generate_cluster_version=10534) [2022-08-08 10:58:26.204125] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:26.204188] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667801, retry interval=1000000, ret=-4179) [2022-08-08 10:58:26.208689] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:26.211851] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:26.211983] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:26.281285] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:26.281415] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] task process failed, start retry(max retry time=9223372036854667806, retry interval=1000000, ret=-4179) [2022-08-08 10:58:26.753690] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11427, time_dist: generate_cluster_version=11423) [2022-08-08 10:58:27.023097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:27.214538] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:27.227282] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:27.227357] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:27.231068] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:27.231118] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667800, retry interval=1000000, ret=-4179) [2022-08-08 10:58:27.264464] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:27.301471] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:27.302574] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=1054] [dc=0] task process failed, start retry(max retry time=9223372036854667805, retry interval=1000000, ret=-4179) [2022-08-08 10:58:28.230143] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:28.234409] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:28.234478] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:28.254596] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:28.254637] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667799, retry interval=1000000, ret=-4179) [2022-08-08 10:58:28.339606] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:28.339770] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667804, retry interval=1000000, ret=-4179) [2022-08-08 10:58:28.936195] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14886) [2022-08-08 10:58:28.943357] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24937, time_dist: do_cluster_heartbeat=12, generate_cluster_version=24920) [2022-08-08 10:58:28.943643] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22114, time_dist: generate_cluster_version=22112) [2022-08-08 10:58:29.139502] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35685) [2022-08-08 10:58:29.142004] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=553] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38947, time_dist: generate_cluster_version=38942) [2022-08-08 10:58:29.142504] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5804] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26968, time_dist: do_cluster_heartbeat=9, generate_cluster_version=26957) [2022-08-08 10:58:29.237255] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:29.240902] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:29.240926] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:29.271462] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:29.271513] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667798, retry interval=1000000, ret=-4179) [2022-08-08 10:58:29.322913] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15653) [2022-08-08 10:58:29.323328] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17063, time_dist: do_cluster_heartbeat=11, generate_cluster_version=17050) [2022-08-08 10:58:29.360375] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:29.360451] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667803, retry interval=1000000, ret=-4179) [2022-08-08 10:58:29.387409] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927509387402, cur_ts_=1659927509366613, delta=20789) [2022-08-08 10:58:29.387456] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27445) [2022-08-08 10:58:29.399551] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=45467, time_dist: generate_cluster_version=45464) [2022-08-08 10:58:29.409132] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19852) [2022-08-08 10:58:29.430137] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20945) [2022-08-08 10:58:29.523316] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15642) [2022-08-08 10:58:29.523580] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17044, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17031) [2022-08-08 10:58:29.575750] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19027) [2022-08-08 10:58:29.576304] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20333, time_dist: generate_cluster_version=20328) [2022-08-08 10:58:29.588826] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927509588817, cur_ts_=1659927509567278, delta=21539) [2022-08-08 10:58:29.724289] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:58:29.724323] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:58:29.771456] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30286) [2022-08-08 10:58:29.771519] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927509771516, cur_ts_=1659927509745334, delta=26182) [2022-08-08 10:58:29.781325] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=41019, time_dist: generate_cluster_version=41015) [2022-08-08 10:58:29.863665] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=92158) [2022-08-08 10:58:29.868658] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=53] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:2592674330102639996, inc:84097144, addr:"192.168.11.128:2882", t:1659927509738416}, log_id=8315950, used_time=127631) [2022-08-08 10:58:29.868857] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=128467, time_dist: generate_cluster_version=128464) [2022-08-08 10:58:29.986031] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] clock out of order(cur_ts=1659927509986015, cur_ts_=1659927509964590, delta=21425) [2022-08-08 10:58:30.032586] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14819616}) [2022-08-08 10:58:30.238956] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927508314686, end_time=1659927510238934, elapsed_time=1924248, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:58:30.244487] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:30.250811] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:30.250841] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:30.317070] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:30.317274] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] task process failed, start retry(max retry time=9223372036854667797, retry interval=1000000, ret=-4179) [2022-08-08 10:58:30.374050] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927510374040, cur_ts_=1659927510341085, delta=32955) [2022-08-08 10:58:30.386535] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46787) [2022-08-08 10:58:30.396111] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=230] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:30.396175] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667802, retry interval=1000000, ret=-4179) [2022-08-08 10:58:30.396279] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=57599, time_dist: generate_cluster_version=57597) [2022-08-08 10:58:30.582120] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:30.588024] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:30.737278] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14160, time_dist: generate_cluster_version=14156) [2022-08-08 10:58:30.774587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:30.844902] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] clock out of order(cur_ts=1659927510844894, cur_ts_=1659927510823558, delta=21336) [2022-08-08 10:58:30.901684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:30.965084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:30.979219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:31.020128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:31.020216] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927511020213, cur_ts_=1659927510856253, delta=163960) [2022-08-08 10:58:31.021471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=161] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:31.065165] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=309] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:31.066617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:31.078825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:31.078999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:31.104869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:31.106838] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12939) [2022-08-08 10:58:31.107121] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19341, time_dist: do_cluster_heartbeat=12, generate_cluster_version=19325) [2022-08-08 10:58:31.108096] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15691, time_dist: do_cluster_heartbeat=11, generate_cluster_version=15677) [2022-08-08 10:58:31.125891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:31.219194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:31.253725] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:31.259950] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:31.259976] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:31.275086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:31.341269] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:31.341333] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667796, retry interval=1000000, ret=-4179) [2022-08-08 10:58:31.417578] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:31.417619] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667801, retry interval=1000000, ret=-4179) [2022-08-08 10:58:31.443738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:31.449607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:31.503175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=256] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:32.265783] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:32.269790] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:32.269815] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:32.364840] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:32.364908] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667795, retry interval=1000000, ret=-4179) [2022-08-08 10:58:32.437414] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:32.437473] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667800, retry interval=1000000, ret=-4179) [2022-08-08 10:58:32.571922] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] clock out of order(cur_ts=1659927512571916, cur_ts_=1659927512542312, delta=29604) [2022-08-08 10:58:32.683728] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:32.720653] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927512720646, cur_ts_=1659927512700224, delta=20422) [2022-08-08 10:58:33.004064] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927513004056, cur_ts_=1659927512983699, delta=20357) [2022-08-08 10:58:33.274410] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:33.277558] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:33.277618] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:33.399761] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:33.401190] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:33.401238] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667794, retry interval=1000000, ret=-4179) [2022-08-08 10:58:33.460789] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:33.460825] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667799, retry interval=1000000, ret=-4179) [2022-08-08 10:58:33.628266] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927513627372 2022-08-08 02:58:33.627372|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:1992002229859640899, inc:84097296, addr:{ip:"192.168.11.128", port:2882}, t:1659927513626000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ABFB1-0-0, start_time:1659927513627255, snapshot:1659927513574781|[end_task] u=180 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927513627570|[calc_checksum_by_commit] u=19 ret:0, checksum:1043985453, checksum_log_ts:9223372036854775807|[submit_log] u=120 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=472 trans_version:1659927513626607, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8315970, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=856) [2022-08-08 10:58:34.250590] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:34.250622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:34.250636] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:34.253383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:34.279818] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:34.283041] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:34.283075] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:34.369564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:34.401018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.422520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:34.422633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:34.423228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.423881] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:34.423942] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667793, retry interval=1000000, ret=-4179) [2022-08-08 10:58:34.481956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:34.482045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:34.483725] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:34.483777] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667798, retry interval=1000000, ret=-4179) [2022-08-08 10:58:34.546168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.580355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:34.587846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=306] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.601847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:34.611635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.613178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:34.637999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:34.715601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.730197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:34.738092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.758143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:34.758214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.761460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:34.769049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:34.793179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.793359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:34.793799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:34.845851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.845943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.846006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:34.862501] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:34.866788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:34.866830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.897911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.897981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:34.899368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:34.899815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:34.904275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.908608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.928879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:34.929373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.929387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.939575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.950166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:34.992432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:34.996018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:34.999999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:35.005520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:35.044626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:35.045144] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.072962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=274] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:35.107288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:35.114198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.185176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.190971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.201503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:35.224160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.259536] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3011-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:35.263773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:35.285761] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:35.288951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:35.288974] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:35.295127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.363567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:35.448164] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:35.448253] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:35.449351] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:35.449412] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667792, retry interval=1000000, ret=-4179) [2022-08-08 10:58:35.461953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.485833] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4ADE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:35.486008] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4ADE-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:35.503832] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:35.503902] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667797, retry interval=1000000, ret=-4179) [2022-08-08 10:58:35.679753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:35.828996] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:58:36.200401] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927516200396, cur_ts_=1659927516164771, delta=35625) [2022-08-08 10:58:36.381708] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:36.384351] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927516384342, cur_ts_=1659927516260946, delta=123396) [2022-08-08 10:58:36.404183] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=143045) [2022-08-08 10:58:36.404280] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=36] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:995889980112373258, inc:84097384, addr:"192.168.11.128:2882", t:1659927516255495}, log_id=8315983, used_time=143199) [2022-08-08 10:58:36.405842] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=150800, time_dist: generate_cluster_version=150795) [2022-08-08 10:58:36.410473] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] slow flush(flush_time=6245, io_time=143076, file_id=237, flush_start_offset=51321831, file_offset=51322447) [2022-08-08 10:58:36.414403] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=10329] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:36.414449] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:36.424826] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=211] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14125) [2022-08-08 10:58:36.425194] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:4734765263742433718, inc:84097385, addr:"192.168.11.128:2882", t:1659927516260946}, log_id=8301392, used_time=161306) [2022-08-08 10:58:36.426302] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=163865, time_dist: do_cluster_heartbeat=14, generate_cluster_version=163846) [2022-08-08 10:58:36.426875] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3014-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:36.483816] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:36.484046] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667791, retry interval=1000000, ret=-4179) [2022-08-08 10:58:36.527860] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:36.528225] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:36.529989] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:36.530061] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667796, retry interval=1000000, ret=-4179) [2022-08-08 10:58:37.043624] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:37.323987] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:37.425478] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:37.430186] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:37.430271] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:37.440300] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:37.440351] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:37.440582] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:37.440904] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:37.511871] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:37.511934] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667790, retry interval=1000000, ret=-4179) [2022-08-08 10:58:37.550540] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:37.550585] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667795, retry interval=1000000, ret=-4179) [2022-08-08 10:58:38.093155] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19070] [dc=0] clock out of order(cur_ts=1659927518093147, cur_ts_=1659927518062287, delta=30860) [2022-08-08 10:58:38.432848] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:38.436927] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:38.436952] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:38.503361] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927518503355, cur_ts_=1659927518475759, delta=27596) [2022-08-08 10:58:38.543550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:38.543745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:38.545249] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:38.545295] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667789, retry interval=1000000, ret=-4179) [2022-08-08 10:58:38.578649] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:38.578704] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667794, retry interval=1000000, ret=-4179) [2022-08-08 10:58:38.753072] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927518753063, cur_ts_=1659927518727175, delta=25888) [2022-08-08 10:58:39.440402] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:39.443148] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:39.443172] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:39.581129] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:39.581160] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667788, retry interval=1000000, ret=-4179) [2022-08-08 10:58:39.608422] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927519608415, cur_ts_=1659927519584727, delta=23688) [2022-08-08 10:58:39.612758] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:39.612849] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667793, retry interval=1000000, ret=-4179) [2022-08-08 10:58:39.730888] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:58:39.730907] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:58:39.955431] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14848992}) [2022-08-08 10:58:40.138650] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927518316377, end_time=1659927520138640, elapsed_time=1822263, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:58:40.444963] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:40.449787] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:40.449844] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:40.477714] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3020-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.477824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3020-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.478183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3020-0-0] [lt=332] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.478196] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3020-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.478644] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.478867] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.478990] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.479120] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:40.598231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:40.603766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:40.615633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4967-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:40.615796] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4967-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:40.623330] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=628] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:40.623870] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=477] [dc=0] task process failed, start retry(max retry time=9223372036854667787, retry interval=1000000, ret=-4179) [2022-08-08 10:58:40.636254] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13276) [2022-08-08 10:58:40.638619] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16847, time_dist: generate_cluster_version=16829) [2022-08-08 10:58:40.651798] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:40.651835] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667792, retry interval=1000000, ret=-4179) [2022-08-08 10:58:40.813213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:40.814123] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927520814116, cur_ts_=1659927520788140, delta=25976) [2022-08-08 10:58:40.828735] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=191] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14584) [2022-08-08 10:58:40.828976] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15565, time_dist: do_cluster_heartbeat=7, generate_cluster_version=15554) [2022-08-08 10:58:40.829423] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16081, time_dist: do_cluster_heartbeat=12, generate_cluster_version=16068) [2022-08-08 10:58:40.922194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:40.980229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.003358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.073129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:41.073326] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=259] [dc=0] clock out of order(cur_ts=1659927521073320, cur_ts_=1659927521040544, delta=32776) [2022-08-08 10:58:41.074688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=341] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:41.076536] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36385) [2022-08-08 10:58:41.076807] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38322, time_dist: generate_cluster_version=38320) [2022-08-08 10:58:41.077283] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=38591, time_dist: generate_cluster_version=38589) [2022-08-08 10:58:41.083112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=1036] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.086686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=2602] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:41.095440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:41.104153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.125828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:41.147168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.236906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:41.292075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.452186] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:41.455249] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:41.455283] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:41.462483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:41.472000] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:41.487102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3023-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:41.487220] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:41.487433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:41.487698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:41.519462] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927521519453, cur_ts_=1659927521492523, delta=26930) [2022-08-08 10:58:41.519696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:41.661247] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24367) [2022-08-08 10:58:41.662298] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32806, time_dist: do_cluster_heartbeat=7, generate_cluster_version=32797) [2022-08-08 10:58:41.664783] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=207] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:41.665667] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667786, retry interval=1000000, ret=-4179) [2022-08-08 10:58:41.676945] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=196] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:41.677014] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667791, retry interval=1000000, ret=-4179) [2022-08-08 10:58:42.008339] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927522008330, cur_ts_=1659927521986928, delta=21402) [2022-08-08 10:58:42.078268] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=18446) [2022-08-08 10:58:42.133950] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=50573) [2022-08-08 10:58:42.134453] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927522134447, cur_ts_=1659927522093459, delta=40988) [2022-08-08 10:58:42.135654] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=76616, time_dist: generate_cluster_version=76613) [2022-08-08 10:58:42.138916] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13948, time_dist: generate_cluster_version=13944) [2022-08-08 10:58:42.457711] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:42.460925] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:42.461054] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:42.694319] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:42.694352] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667785, retry interval=1000000, ret=-4179) [2022-08-08 10:58:42.709385] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:42.709435] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667790, retry interval=1000000, ret=-4179) [2022-08-08 10:58:43.327922] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927523327912, cur_ts_=1659927523306761, delta=21151) [2022-08-08 10:58:43.463434] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:43.467809] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:43.467904] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:43.717247] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:43.717751] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] task process failed, start retry(max retry time=9223372036854667784, retry interval=1000000, ret=-4179) [2022-08-08 10:58:43.732088] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:43.732150] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667789, retry interval=1000000, ret=-4179) [2022-08-08 10:58:44.086156] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11223) [2022-08-08 10:58:44.086416] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16049, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16036) [2022-08-08 10:58:44.122798] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31261) [2022-08-08 10:58:44.124283] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=2154] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37733, time_dist: generate_cluster_version=37728) [2022-08-08 10:58:44.277303] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:44.388115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:44.421314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.442474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.517169] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:44.521540] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2628] [dc=0] clock out of order(cur_ts=1659927524521531, cur_ts_=1659927524448001, delta=73530) [2022-08-08 10:58:44.569233] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43111) [2022-08-08 10:58:44.574515] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:44.574552] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:44.585380] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.604006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:44.610474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.620855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:44.640062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.640107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:44.653951] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=149215, time_dist: do_cluster_heartbeat=13, generate_cluster_version=149197) [2022-08-08 10:58:44.655501] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:44.656037] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=86759) [2022-08-08 10:58:44.682989] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=23] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:10388231734365081787, inc:84097696, addr:"192.168.11.128:2882", t:1659927524448001}, log_id=8301433, used_time=172222) [2022-08-08 10:58:44.684186] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=174602, time_dist: do_cluster_heartbeat=12, generate_cluster_version=174587) [2022-08-08 10:58:44.698316] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=166] [dc=0] clock out of order(cur_ts=1659927524698306, cur_ts_=1659927524654189, delta=44117) [2022-08-08 10:58:44.736857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.748241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:44.775392] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=58178) [2022-08-08 10:58:44.776041] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=60464, time_dist: do_cluster_heartbeat=9, generate_cluster_version=60452) [2022-08-08 10:58:44.777992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.778336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:44.778678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:44.778719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.785629] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:44.826790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=568] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.832560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:44.832668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=405] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:44.843595] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927524843586, cur_ts_=1659927524747925, delta=95661) [2022-08-08 10:58:44.886637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:44.886851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.887014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:44.887478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.887581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.887642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:44.913804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.913881] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=138319) [2022-08-08 10:58:44.913920] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] slow flush(flush_time=29, io_time=138328, file_id=237, flush_start_offset=51427076, file_offset=51427698) [2022-08-08 10:58:44.922784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:44.923499] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:10850590872992399711, inc:84097703, addr:"192.168.11.128:2882", t:1659927524737113}, log_id=8301434, used_time=186145) [2022-08-08 10:58:44.923833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.924943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:44.926558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.926701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:44.935696] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21755) [2022-08-08 10:58:44.937084] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:10847451825722849460, inc:84097697, addr:"192.168.11.128:2882", t:1659927524532698}, log_id=1655736, used_time=154438) [2022-08-08 10:58:44.939969] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=203287, time_dist: do_cluster_heartbeat=11, generate_cluster_version=203270) [2022-08-08 10:58:44.969098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.969162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.969181] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] clock out of order(cur_ts=1659927524969176, cur_ts_=1659927524941061, delta=28115) [2022-08-08 10:58:44.969328] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:44.969570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:44.969652] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:44.997322] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:44.997360] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667788, retry interval=1000000, ret=-4179) [2022-08-08 10:58:45.004356] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=66225) [2022-08-08 10:58:45.011045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:45.025991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:45.026055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:45.026682] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.042284] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37873) [2022-08-08 10:58:45.060851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.072747] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927525072740, cur_ts_=1659927525037068, delta=35672) [2022-08-08 10:58:45.068560] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30164] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:45.073398] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667783, retry interval=1000000, ret=-4179) [2022-08-08 10:58:45.109203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:45.072379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:45.113449] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=110001, time_dist: do_cluster_heartbeat=8, generate_cluster_version=109988) [2022-08-08 10:58:45.120397] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=140245, time_dist: do_cluster_heartbeat=11, generate_cluster_version=140229) [2022-08-08 10:58:45.128059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:45.133865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.218373] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927525218362, cur_ts_=1659927525180321, delta=38041) [2022-08-08 10:58:45.219204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.219603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:45.220047] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30260] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39211) [2022-08-08 10:58:45.229882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.230948] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=51671, time_dist: do_cluster_heartbeat=11, generate_cluster_version=51657) [2022-08-08 10:58:45.241505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.280860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:45.317174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.326411] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5438] [dc=0] clock out of order(cur_ts=1659927525326405, cur_ts_=1659927525300717, delta=25688) [2022-08-08 10:58:45.404669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:45.452689] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=697] [dc=0] clock out of order(cur_ts=1659927525452680, cur_ts_=1659927525410911, delta=41769) [2022-08-08 10:58:45.484244] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.534924] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13434, time_dist: do_cluster_heartbeat=11, generate_cluster_version=13419) [2022-08-08 10:58:45.627621] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:45.631962] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:45.632008] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:45.659953] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927525659947, cur_ts_=1659927525634756, delta=25191) [2022-08-08 10:58:45.699169] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:45.768094] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] clock out of order(cur_ts=1659927525768085, cur_ts_=1659927525745181, delta=22904) [2022-08-08 10:58:45.768289] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30008) [2022-08-08 10:58:45.768553] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31125, time_dist: do_cluster_heartbeat=12, generate_cluster_version=31110) [2022-08-08 10:58:45.769259] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30951, time_dist: print_stat=5, do_cluster_heartbeat=2, generate_cluster_version=30937, cluster_service_self_check=4) [2022-08-08 10:58:45.997534] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927525997527, cur_ts_=1659927525955360, delta=42167) [2022-08-08 10:58:46.006567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE302F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.006627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE302F-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.061382] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE302F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.061416] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE302F-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.061556] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.061576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.061597] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:46.099453] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927526099444, cur_ts_=1659927526006541, delta=92903) [2022-08-08 10:58:46.236415] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927526236410, cur_ts_=1659927526141667, delta=94743) [2022-08-08 10:58:46.279485] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=338731) [2022-08-08 10:58:46.279523] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] slow flush(flush_time=16, io_time=338760, file_id=237, flush_start_offset=51438672, file_offset=51439294) [2022-08-08 10:58:46.279859] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:13683996339295130024, inc:84097742, addr:"192.168.11.128:2882", t:1659927525939254}, log_id=8301440, used_time=339148) [2022-08-08 10:58:46.410050] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927526410033, cur_ts_=1659927526236410, delta=173623) [2022-08-08 10:58:46.416408] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=476722, time_dist: do_cluster_heartbeat=41, generate_cluster_version=476676) [2022-08-08 10:58:46.431187] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=151643) [2022-08-08 10:58:46.431230] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] slow flush(flush_time=23, io_time=151668, file_id=237, flush_start_offset=51439294, file_offset=51439910) [2022-08-08 10:58:46.435774] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=17] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:1590945890300322198, inc:84097743, addr:"192.168.11.128:2882", t:1659927525942050}, log_id=8316032, used_time=492553) [2022-08-08 10:58:46.449992] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:46.450048] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667787, retry interval=1000000, ret=-4179) [2022-08-08 10:58:46.450273] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=507766, time_dist: generate_cluster_version=507764) [2022-08-08 10:58:46.479014] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:46.479055] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667782, retry interval=1000000, ret=-4179) [2022-08-08 10:58:46.479293] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26843) [2022-08-08 10:58:46.505586] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26229) [2022-08-08 10:58:46.514457] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] clock out of order(cur_ts=1659927526514448, cur_ts_=1659927526481994, delta=32454) [2022-08-08 10:58:46.515590] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43164, time_dist: do_cluster_heartbeat=51, generate_cluster_version=43110) [2022-08-08 10:58:46.528489] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13300, time_dist: do_cluster_heartbeat=11, generate_cluster_version=13283) [2022-08-08 10:58:46.630173] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27622) [2022-08-08 10:58:46.630558] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=45199, time_dist: generate_cluster_version=45196) [2022-08-08 10:58:46.674173] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:46.674280] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927526674274, cur_ts_=1659927526651238, delta=23036) [2022-08-08 10:58:46.695949] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:46.695982] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:46.774516] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927526774506, cur_ts_=1659927526742204, delta=32302) [2022-08-08 10:58:46.906111] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927526906104, cur_ts_=1659927526881290, delta=24814) [2022-08-08 10:58:47.010573] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927527010562, cur_ts_=1659927526989417, delta=21145) [2022-08-08 10:58:47.029099] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=50525) [2022-08-08 10:58:47.067534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:47.076550] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=105517, time_dist: print_stat=4, do_cluster_heartbeat=2, generate_cluster_version=105500, cluster_service_self_check=7) [2022-08-08 10:58:47.123063] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927527123054, cur_ts_=1659927527096648, delta=26406) [2022-08-08 10:58:47.199073] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=71278) [2022-08-08 10:58:47.291929] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927527291922, cur_ts_=1659927527253889, delta=38033) [2022-08-08 10:58:47.315037] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=66865) [2022-08-08 10:58:47.315381] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=74785, time_dist: generate_cluster_version=74783) [2022-08-08 10:58:47.354713] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39630) [2022-08-08 10:58:47.355139] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=10] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:9449746850296549533, inc:84097787, addr:"192.168.11.128:2882", t:1659927527250908}, log_id=8316037, used_time=103686) [2022-08-08 10:58:47.371048] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=120354, time_dist: do_cluster_heartbeat=8, generate_cluster_version=120341) [2022-08-08 10:58:47.430296] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927527430288, cur_ts_=1659927527390893, delta=39395) [2022-08-08 10:58:47.455357] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19240) [2022-08-08 10:58:47.470336] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:47.479192] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=44022, time_dist: do_cluster_heartbeat=7, generate_cluster_version=44011) [2022-08-08 10:58:47.484757] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29327) [2022-08-08 10:58:47.530740] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=94447, time_dist: print_stat=6, do_cluster_heartbeat=3, generate_cluster_version=94430, cluster_service_self_check=5) [2022-08-08 10:58:47.544805] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3867] [dc=0] clock out of order(cur_ts=1659927527544796, cur_ts_=1659927527501044, delta=43752) [2022-08-08 10:58:47.581436] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=92548) [2022-08-08 10:58:47.582953] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] transaction log sync use too much time(partition={tid:1099511628085, partition_id:0, part_cnt:0}, trans_id={hash:6840715160930807054, inc:84097795, addr:"192.168.11.128:2882", t:1659927527450999}, log_id=1160621, used_time=111621) [2022-08-08 10:58:47.597668] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16111) [2022-08-08 10:58:47.654501] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=106737, time_dist: do_cluster_heartbeat=9, generate_cluster_version=106726) [2022-08-08 10:58:47.670220] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927527670213, cur_ts_=1659927527632060, delta=38153) [2022-08-08 10:58:47.670861] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:47.670898] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667781, retry interval=1000000, ret=-4179) [2022-08-08 10:58:47.681912] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=84156) [2022-08-08 10:58:47.701460] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=114461, time_dist: do_cluster_heartbeat=10, generate_cluster_version=114447) [2022-08-08 10:58:47.720439] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:47.720480] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=3072] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35468) [2022-08-08 10:58:47.729287] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:47.729598] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667786, retry interval=1000000, ret=-4179) [2022-08-08 10:58:47.736605] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16088) [2022-08-08 10:58:47.739626] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:47.739649] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:47.765889] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29245) [2022-08-08 10:58:47.766084] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=45733, time_dist: do_cluster_heartbeat=13, generate_cluster_version=45718) [2022-08-08 10:58:47.899661] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23077, time_dist: generate_cluster_version=23074) [2022-08-08 10:58:47.940627] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927527940617, cur_ts_=1659927527909602, delta=31015) [2022-08-08 10:58:48.061779] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30950) [2022-08-08 10:58:48.062160] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32298, time_dist: generate_cluster_version=32295) [2022-08-08 10:58:48.063190] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16256, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16237, cluster_service_self_check=5) [2022-08-08 10:58:48.205293] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] clock out of order(cur_ts=1659927528205286, cur_ts_=1659927528163525, delta=41761) [2022-08-08 10:58:48.280707] WARN [CLOG] submit_log_succ_cb (ob_log_task.cpp:323) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] transaction on success costs(on_success_duration=26841, pkey={tid:1100611139404002, partition_id:0, part_cnt:0}, log_id=8301450) [2022-08-08 10:58:48.280836] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29209, time_dist: do_cluster_heartbeat=10, generate_cluster_version=29197) [2022-08-08 10:58:48.305453] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=51778) [2022-08-08 10:58:48.305656] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=53507, time_dist: do_cluster_heartbeat=6, generate_cluster_version=53499) [2022-08-08 10:58:48.398282] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] clock out of order(cur_ts=1659927528398273, cur_ts_=1659927528365130, delta=33143) [2022-08-08 10:58:48.543669] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] clock out of order(cur_ts=1659927528543659, cur_ts_=1659927528502063, delta=41596) [2022-08-08 10:58:48.587909] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=7591] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=87244) [2022-08-08 10:58:48.588146] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=88435, time_dist: do_cluster_heartbeat=11, generate_cluster_version=88420) [2022-08-08 10:58:48.650242] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927528650234, cur_ts_=1659927528547486, delta=102748) [2022-08-08 10:58:48.785897] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927528785886, cur_ts_=1659927528746404, delta=39482) [2022-08-08 10:58:48.809872] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:48.856271] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=147749) [2022-08-08 10:58:48.857222] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:14754981527199787993, inc:84097852, addr:"192.168.11.128:2882", t:1659927528650234}, log_id=8301452, used_time=148757) [2022-08-08 10:58:48.857384] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=149579, time_dist: do_cluster_heartbeat=9, generate_cluster_version=149569) [2022-08-08 10:58:48.877164] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] slow flush(flush_time=20869, io_time=147774, file_id=237, flush_start_offset=51464602, file_offset=51465224) [2022-08-08 10:58:48.887855] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:48.887876] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:48.889007] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] clock out of order(cur_ts=1659927528889002, cur_ts_=1659927528855676, delta=33326) [2022-08-08 10:58:48.910083] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32890) [2022-08-08 10:58:48.910298] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=160455, time_dist: do_cluster_heartbeat=10, generate_cluster_version=160443) [2022-08-08 10:58:48.933578] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23442) [2022-08-08 10:58:48.938616] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=10] [dc=0] [trans sampling] (TRACE=begin_ts=1659927528909366 2022-08-08 02:58:48.909366|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:7761047444657503654, inc:84097860, addr:{ip:"192.168.11.128", port:2882}, t:1659927528903000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A622F-0-0, start_time:1659927528909217, snapshot:1659927528900037|[end_task] u=247 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927528909632|[calc_checksum_by_commit] u=23 ret:0, checksum:216698733, checksum_log_ts:9223372036854775807|[submit_log] u=28 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=28823 trans_version:1659927528909657, uref:1073741825|[exiting] u=29 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=5 ret:0, log_type:64, log_id:8301453, logging:0, pending:0, uref:1|[end_trans_cb] u=12 ret:0, param:0, param:true|[destroy] u=11 arg1:140085232375600|total_timeu=29202) [2022-08-08 10:58:48.944064] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35206, time_dist: do_cluster_heartbeat=11, generate_cluster_version=35193) [2022-08-08 10:58:48.952064] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:48.952104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667785, retry interval=1000000, ret=-4179) [2022-08-08 10:58:48.970561] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:48.970597] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667780, retry interval=1000000, ret=-4179) [2022-08-08 10:58:48.972719] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18784) [2022-08-08 10:58:48.991688] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18915) [2022-08-08 10:58:48.998660] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=33692, time_dist: do_cluster_heartbeat=9, generate_cluster_version=33679) [2022-08-08 10:58:49.149798] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16463) [2022-08-08 10:58:49.149986] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18089, time_dist: generate_cluster_version=18087) [2022-08-08 10:58:49.550262] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3038-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:49.550415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:49.550526] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:49.550656] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:49.550708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:49.731002] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:58:49.731034] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:58:49.911470] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:49.913300] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21832) [2022-08-08 10:58:49.913552] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22804, time_dist: generate_cluster_version=22802) [2022-08-08 10:58:49.919141] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:49.919187] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:49.955511] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.955694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.955780] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.955816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.955972] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.956024] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.956192] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.956400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.956526] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.956720] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.956874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49AF-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:49.983510] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927529983504, cur_ts_=1659927529961365, delta=22139) [2022-08-08 10:58:49.990319] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:49.990886] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] task process failed, start retry(max retry time=9223372036854667779, retry interval=1000000, ret=-4179) [2022-08-08 10:58:50.043124] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=50409) [2022-08-08 10:58:50.049286] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:50.049348] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667784, retry interval=1000000, ret=-4179) [2022-08-08 10:58:50.057338] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14175) [2022-08-08 10:58:50.058020] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22377, time_dist: print_stat=4, do_cluster_heartbeat=3, generate_cluster_version=22366) [2022-08-08 10:58:50.058260] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23541, time_dist: generate_cluster_version=23539) [2022-08-08 10:58:50.211702] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927530211698, cur_ts_=1659927530190913, delta=20785) [2022-08-08 10:58:50.259235] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11729) [2022-08-08 10:58:50.260077] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35041, time_dist: do_cluster_heartbeat=2061, generate_cluster_version=32977) [2022-08-08 10:58:50.337249] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31068) [2022-08-08 10:58:50.338059] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=33704, time_dist: generate_cluster_version=33700) [2022-08-08 10:58:50.442122] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927530442113, cur_ts_=1659927530418212, delta=23901) [2022-08-08 10:58:50.470695] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11408) [2022-08-08 10:58:50.473669] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14855104}) [2022-08-08 10:58:50.473728] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20018, time_dist: do_cluster_heartbeat=15, generate_cluster_version=20001) [2022-08-08 10:58:50.494340] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=39424, time_dist: do_cluster_heartbeat=10, generate_cluster_version=39409) [2022-08-08 10:58:50.558868] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927530558859, cur_ts_=1659927530523845, delta=35014) [2022-08-08 10:58:50.623433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:50.623585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:50.732778] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927528321691, end_time=1659927530732766, elapsed_time=2411075, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:58:50.836181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=1118] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:50.920874] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:50.923993] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:50.924032] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:50.943164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:50.983230] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927530983223, cur_ts_=1659927530959032, delta=24191) [2022-08-08 10:58:50.995057] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.995161] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.995198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.995218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.995490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.995862] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.995916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:50.996016] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4AF3-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:51.008151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.015876] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:51.015926] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667778, retry interval=1000000, ret=-4179) [2022-08-08 10:58:51.028531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.068344] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:51.068395] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667783, retry interval=1000000, ret=-4179) [2022-08-08 10:58:51.089981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:51.090027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:51.100403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.102618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:51.120708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:51.122878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.143202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:51.164076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.258242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:51.316752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.492774] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927531492764, cur_ts_=1659927531463531, delta=29233) [2022-08-08 10:58:51.494802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:51.494914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:51.495273] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32965) [2022-08-08 10:58:51.495459] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7811] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35084, time_dist: generate_cluster_version=35082) [2022-08-08 10:58:51.536949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:51.926485] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:51.930690] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:51.930783] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:52.018881] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49B2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:52.018921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49B2-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:52.021214] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:52.021465] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:52.039862] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:52.039915] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667777, retry interval=1000000, ret=-4179) [2022-08-08 10:58:52.089247] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927532089241, cur_ts_=1659927532067900, delta=21341) [2022-08-08 10:58:52.110639] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:52.110744] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667782, retry interval=1000000, ret=-4179) [2022-08-08 10:58:52.489509] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927532489500, cur_ts_=1659927532453722, delta=35778) [2022-08-08 10:58:52.510502] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10523, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10511) [2022-08-08 10:58:52.512451] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11749, time_dist: do_cluster_heartbeat=5, generate_cluster_version=11741) [2022-08-08 10:58:52.683894] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:52.948576] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:52.948588] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927532948582, cur_ts_=1659927532920901, delta=27681) [2022-08-08 10:58:52.952732] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:52.952816] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:53.042616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.042661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.042720] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.042816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.043008] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.043120] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.043296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.043598] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.043822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.044016] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.044041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6180-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:53.066775] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:53.066806] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667776, retry interval=1000000, ret=-4179) [2022-08-08 10:58:53.130058] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:53.130095] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667781, retry interval=1000000, ret=-4179) [2022-08-08 10:58:53.737849] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927533737839, cur_ts_=1659927533706409, delta=31430) [2022-08-08 10:58:53.846053] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1183] [dc=0] clock out of order(cur_ts=1659927533846044, cur_ts_=1659927533824139, delta=21905) [2022-08-08 10:58:53.955080] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:53.958545] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:53.958584] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:54.094239] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927534094234, cur_ts_=1659927534060196, delta=34038) [2022-08-08 10:58:54.187948] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:54.187991] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667775, retry interval=1000000, ret=-4179) [2022-08-08 10:58:54.189215] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19801, time_dist: generate_cluster_version=19797) [2022-08-08 10:58:54.189259] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17700) [2022-08-08 10:58:54.192540] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23753, time_dist: do_cluster_heartbeat=8, generate_cluster_version=23740) [2022-08-08 10:58:54.200425] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:54.200480] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667780, retry interval=1000000, ret=-4179) [2022-08-08 10:58:54.297313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=128] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:54.403757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:54.439180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.458837] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.607194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.628675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.629502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:54.646168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:54.656825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.656940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:54.671914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:54.754151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.771630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:54.796916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.797826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:54.801268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:54.802430] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.807409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:54.849223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.850127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:54.852371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:54.914223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:58:54.915416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:54.915698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:54.921005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.919604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.919648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.931751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.941196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:54.941635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.949079] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:54.950172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.950291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:54.962013] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:54.970772] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=338] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:54.970967] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:54.985365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:54.985800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.985828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:54.986125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:54.994102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:55.012460] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1418] [dc=0] clock out of order(cur_ts=1659927535012455, cur_ts_=1659927534991401, delta=21054) [2022-08-08 10:58:55.030800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:55.045315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:55.077617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:58:55.079077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=11759] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.080228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=15622] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.091232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=40976] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:55.133640] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:58:55.144043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:55.154916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.190110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.190165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.190251] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.190302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.190453] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.190518] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.190749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.191038] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.191265] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.191277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6183-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:55.227750] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:55.227880] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667774, retry interval=1000000, ret=-4179) [2022-08-08 10:58:55.237637] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:55.237693] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667779, retry interval=1000000, ret=-4179) [2022-08-08 10:58:55.239294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=8276] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.239573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=7953] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:55.254067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.260226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.304628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=504] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:58:55.335451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.429557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:58:55.505621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.603764] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927535603757, cur_ts_=1659927535570185, delta=33572) [2022-08-08 10:58:55.719512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:58:55.857374] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:58:55.973817] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:55.978445] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:55.978485] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:56.230084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.230129] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.230226] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.230463] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.230852] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=357] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.230877] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.230889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.231614] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C497D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:56.254872] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:56.254922] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667773, retry interval=1000000, ret=-4179) [2022-08-08 10:58:56.260100] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:56.260159] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667778, retry interval=1000000, ret=-4179) [2022-08-08 10:58:56.622610] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:56.622698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:56.622719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:56.980697] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:56.984923] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:56.985085] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:57.089705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:58:57.261346] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6186-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:57.266202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:58:57.288031] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:57.288066] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667777, retry interval=1000000, ret=-4179) [2022-08-08 10:58:57.293129] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:57.293176] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667772, retry interval=1000000, ret=-4179) [2022-08-08 10:58:57.640448] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3050-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:57.640492] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3050-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:57.640639] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3050-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:57.640925] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:57.641078] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:57.837476] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:57.987971] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:57.991080] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:57.991115] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:58.185381] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=56750, time_dist: do_cluster_heartbeat=56732, generate_cluster_version=14) [2022-08-08 10:58:58.232236] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927538232227, cur_ts_=1659927538109501, delta=122726) [2022-08-08 10:58:58.301122] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18400) [2022-08-08 10:58:58.306090] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=323] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=42985, time_dist: do_cluster_heartbeat=7, generate_cluster_version=42975) [2022-08-08 10:58:58.376506] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927538376497, cur_ts_=1659927538335675, delta=40822) [2022-08-08 10:58:58.416603] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=115404) [2022-08-08 10:58:58.416639] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] slow flush(flush_time=18, io_time=115428, file_id=237, flush_start_offset=51559865, file_offset=51560481) [2022-08-08 10:58:58.419249] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=31] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:16401638203141415417, inc:84098216, addr:"192.168.11.128:2882", t:1659927538268044}, log_id=8316095, used_time=150759) [2022-08-08 10:58:58.420154] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=156235, time_dist: do_cluster_heartbeat=4, generate_cluster_version=156228) [2022-08-08 10:58:58.436952] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20294) [2022-08-08 10:58:58.437756] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=55484, time_dist: do_cluster_heartbeat=49, generate_cluster_version=55432) [2022-08-08 10:58:58.454273] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:58.454317] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667776, retry interval=1000000, ret=-4179) [2022-08-08 10:58:58.459758] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:58.459797] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667771, retry interval=1000000, ret=-4179) [2022-08-08 10:58:58.490998] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3879] [dc=0] clock out of order(cur_ts=1659927538490990, cur_ts_=1659927538470599, delta=20391) [2022-08-08 10:58:58.650148] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3053-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:58.650254] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:58:58.938087] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11682, time_dist: do_cluster_heartbeat=9, generate_cluster_version=11669) [2022-08-08 10:58:58.995852] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:58:58.999027] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:58:58.999051] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:58:59.286422] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927539286416, cur_ts_=1659927539261476, delta=24940) [2022-08-08 10:58:59.328506] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10529, time_dist: generate_cluster_version=10527) [2022-08-08 10:58:59.355691] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15931) [2022-08-08 10:58:59.356063] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17322, time_dist: generate_cluster_version=17320) [2022-08-08 10:58:59.384542] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20333) [2022-08-08 10:58:59.516934] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:59.517007] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667770, retry interval=1000000, ret=-4179) [2022-08-08 10:58:59.524164] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:58:59.524200] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667775, retry interval=1000000, ret=-4179) [2022-08-08 10:58:59.541750] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10362) [2022-08-08 10:58:59.732729] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:58:59.732746] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:58:59.966543] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927539966533, cur_ts_=1659927539943734, delta=22799) [2022-08-08 10:59:00.000773] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:00.003950] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:00.003977] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:00.121672] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27745) [2022-08-08 10:59:00.122125] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29848, time_dist: do_cluster_heartbeat=18, generate_cluster_version=29827) [2022-08-08 10:59:00.123961] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32614, time_dist: do_cluster_heartbeat=9, generate_cluster_version=32600) [2022-08-08 10:59:00.147933] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14859520}) [2022-08-08 10:59:00.361401] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=262] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927538338924, end_time=1659927540361386, elapsed_time=2022462, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:59:00.519501] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4983-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:00.519553] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4983-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:00.519729] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4983-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:00.519834] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4983-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:00.519903] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4983-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:00.520073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4983-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:00.557939] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:00.557976] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667769, retry interval=1000000, ret=-4179) [2022-08-08 10:59:00.574766] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:00.574806] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667774, retry interval=1000000, ret=-4179) [2022-08-08 10:59:00.574755] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24833) [2022-08-08 10:59:00.575377] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26515, time_dist: generate_cluster_version=26513) [2022-08-08 10:59:00.576192] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20751, time_dist: generate_cluster_version=20749) [2022-08-08 10:59:00.642107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:00.646271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:00.660943] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:00.661098] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:00.661254] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:00.661400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:00.854347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:00.967480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:01.025025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.033448] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:01.037178] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:01.037219] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:01.054991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.118378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:01.119070] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927541119065, cur_ts_=1659927541076125, delta=42940) [2022-08-08 10:59:01.123363] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.123894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=652] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:01.143880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:01.163223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:01.192282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.220556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.220712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:01.252784] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927541252769, cur_ts_=1659927541220993, delta=31776) [2022-08-08 10:59:01.275338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:01.323643] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=53213) [2022-08-08 10:59:01.326608] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=58063, time_dist: do_cluster_heartbeat=9, generate_cluster_version=58049) [2022-08-08 10:59:01.331262] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12979, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12965) [2022-08-08 10:59:01.334344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.409355] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927541409345, cur_ts_=1659927541387542, delta=21803) [2022-08-08 10:59:01.449879] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14262) [2022-08-08 10:59:01.510012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:01.511123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:01.558268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:01.565156] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565178] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565203] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.565339] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC618C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:01.587039] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24331) [2022-08-08 10:59:01.587647] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25738, time_dist: do_cluster_heartbeat=10, generate_cluster_version=25725) [2022-08-08 10:59:01.624573] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30684) [2022-08-08 10:59:01.634883] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:01.634924] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667768, retry interval=1000000, ret=-4179) [2022-08-08 10:59:01.646052] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=83330, time_dist: generate_cluster_version=83326) [2022-08-08 10:59:01.664343] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21889) [2022-08-08 10:59:01.676230] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:01.676288] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667773, retry interval=1000000, ret=-4179) [2022-08-08 10:59:01.871253] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927541871247, cur_ts_=1659927541843940, delta=27307) [2022-08-08 10:59:01.977180] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927541977170, cur_ts_=1659927541945202, delta=31968) [2022-08-08 10:59:02.049127] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:02.055159] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:02.055196] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:02.227187] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927542227177, cur_ts_=1659927542206232, delta=20945) [2022-08-08 10:59:02.440560] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] clock out of order(cur_ts=1659927542440550, cur_ts_=1659927542411913, delta=28637) [2022-08-08 10:59:02.637121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637158] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637290] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637600] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=52] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637678] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637799] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637828] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.637901] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4986-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:02.670073] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=439] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:02.670401] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] task process failed, start retry(max retry time=9223372036854667767, retry interval=1000000, ret=-4179) [2022-08-08 10:59:02.671709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE305F-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.671748] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE305F-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.671940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE305F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.672406] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE305F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.672532] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.672570] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.672587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.672737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:02.707646] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:02.707722] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667772, retry interval=1000000, ret=-4179) [2022-08-08 10:59:03.057663] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:03.080153] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927543080144, cur_ts_=1659927543059385, delta=20759) [2022-08-08 10:59:03.081964] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:03.082000] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:03.561546] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 10:59:03.672189] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B05-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:03.700171] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:03.700221] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667766, retry interval=1000000, ret=-4179) [2022-08-08 10:59:03.738725] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:03.738776] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667771, retry interval=1000000, ret=-4179) [2022-08-08 10:59:03.763723] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12937) [2022-08-08 10:59:03.766302] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16274, time_dist: print_stat=8, generate_cluster_version=16261) [2022-08-08 10:59:03.768245] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12690, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12676) [2022-08-08 10:59:03.964940] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927543964934, cur_ts_=1659927543940967, delta=23967) [2022-08-08 10:59:04.084339] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:04.087461] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:04.087499] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:04.320022] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:04.427111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:04.457366] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.477780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.641107] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927544641102, cur_ts_=1659927544601721, delta=39381) [2022-08-08 10:59:04.641990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.652151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.652347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:04.663576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:04.672301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:04.672395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.699099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:04.707148] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49C4-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:04.707169] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49C4-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:04.707269] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49C4-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:04.719119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49C4-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:04.719182] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49C4-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:04.774356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.777034] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24710) [2022-08-08 10:59:04.778294] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:04.779657] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667765, retry interval=1000000, ret=-4179) [2022-08-08 10:59:04.779727] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927544779721, cur_ts_=1659927544755407, delta=24314) [2022-08-08 10:59:04.785995] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34504, time_dist: do_cluster_heartbeat=6, generate_cluster_version=34492) [2022-08-08 10:59:04.787374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:04.799833] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=200] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:04.799903] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667770, retry interval=1000000, ret=-4179) [2022-08-08 10:59:04.817194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:04.817194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.817340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:04.817756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.829495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:04.874063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.874300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:04.874390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:04.936559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=2696] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.936786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=2691] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.938487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=6640] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:04.939199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=5417] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:04.939272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=5178] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:04.946213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.949398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.956825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:04.959352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:04.967394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:04.969370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:04.971251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.001491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:05.001715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:05.010480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:05.010615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:05.012081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:05.056215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=411] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:05.060431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=203] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:05.089751] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:05.092993] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:05.093180] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:05.095806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.096034] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:05.097741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.106458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:05.156289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:05.168184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:05.179553] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.256687] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.260544] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:05.271915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.284303] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.326374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:05.360579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.448050] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:05.521248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.597065] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12950) [2022-08-08 10:59:05.597343] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13976, time_dist: generate_cluster_version=13974) [2022-08-08 10:59:05.741547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:05.782018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B08-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:05.782052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B08-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:05.782059] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B08-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:05.782202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B08-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:05.782381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B08-0-0] [lt=53] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:05.800372] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:05.800408] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667764, retry interval=1000000, ret=-4179) [2022-08-08 10:59:05.825312] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:05.825347] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667769, retry interval=1000000, ret=-4179) [2022-08-08 10:59:06.096871] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:06.101871] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:06.101970] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:06.392646] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927546392636, cur_ts_=1659927546369170, delta=23466) [2022-08-08 10:59:06.395061] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25906) [2022-08-08 10:59:06.395344] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27391, time_dist: do_cluster_heartbeat=12, generate_cluster_version=27375) [2022-08-08 10:59:06.521252] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927546521243, cur_ts_=1659927546486437, delta=34806) [2022-08-08 10:59:06.800291] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=578] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12673) [2022-08-08 10:59:06.800656] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13730, time_dist: generate_cluster_version=13728) [2022-08-08 10:59:06.800669] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13892, time_dist: do_cluster_heartbeat=9, generate_cluster_version=13881) [2022-08-08 10:59:06.823938] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:06.823996] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667763, retry interval=1000000, ret=-4179) [2022-08-08 10:59:06.852417] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:06.852457] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667768, retry interval=1000000, ret=-4179) [2022-08-08 10:59:07.109510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:07.114917] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:07.118773] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:07.118800] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:07.844229] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:07.844365] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667762, retry interval=1000000, ret=-4179) [2022-08-08 10:59:07.878923] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:07.879069] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] task process failed, start retry(max retry time=9223372036854667767, retry interval=1000000, ret=-4179) [2022-08-08 10:59:07.905108] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:08.121315] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:08.124767] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:08.124793] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:08.464851] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927548464842, cur_ts_=1659927548437764, delta=27078) [2022-08-08 10:59:08.632978] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11031) [2022-08-08 10:59:08.633406] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12425, time_dist: generate_cluster_version=12421) [2022-08-08 10:59:08.651774] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18598) [2022-08-08 10:59:08.651997] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27750, time_dist: generate_cluster_version=27748) [2022-08-08 10:59:08.804674] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12080) [2022-08-08 10:59:08.813836] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23804, time_dist: generate_cluster_version=23800) [2022-08-08 10:59:08.864186] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:08.864237] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667761, retry interval=1000000, ret=-4179) [2022-08-08 10:59:08.903839] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:08.903890] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667766, retry interval=1000000, ret=-4179) [2022-08-08 10:59:08.952569] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927548952559, cur_ts_=1659927548926708, delta=25851) [2022-08-08 10:59:09.039977] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10205, time_dist: generate_cluster_version=10202) [2022-08-08 10:59:09.041221] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10713, time_dist: do_cluster_heartbeat=6, generate_cluster_version=10703) [2022-08-08 10:59:09.127991] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:09.132082] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:09.132112] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:09.549768] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11411) [2022-08-08 10:59:09.621534] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927549621530, cur_ts_=1659927549589178, delta=32352) [2022-08-08 10:59:09.726887] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927549726879, cur_ts_=1659927549692527, delta=34352) [2022-08-08 10:59:09.730950] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=42663, time_dist: generate_cluster_version=42660) [2022-08-08 10:59:09.734100] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:59:09.734132] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:59:09.879438] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15863, time_dist: generate_cluster_version=15860) [2022-08-08 10:59:09.899802] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:09.902410] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] task process failed, start retry(max retry time=9223372036854667760, retry interval=1000000, ret=-4179) [2022-08-08 10:59:09.923601] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21919) [2022-08-08 10:59:09.945389] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:09.945461] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667765, retry interval=1000000, ret=-4179) [2022-08-08 10:59:09.958563] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10229) [2022-08-08 10:59:10.076386] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927550076378, cur_ts_=1659927550043093, delta=33285) [2022-08-08 10:59:10.085399] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10701, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10685, cluster_service_self_check=3) [2022-08-08 10:59:10.123032] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14863968}) [2022-08-08 10:59:10.155238] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:10.160594] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:10.160622] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:10.220763] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10819) [2022-08-08 10:59:10.220983] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11808, time_dist: generate_cluster_version=11805) [2022-08-08 10:59:10.347450] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927548336533, end_time=1659927550347435, elapsed_time=2010902, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:59:10.666164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:10.669644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=155] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:10.872470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:10.924163] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:10.924227] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667759, retry interval=1000000, ret=-4179) [2022-08-08 10:59:10.964373] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:10.964424] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667764, retry interval=1000000, ret=-4179) [2022-08-08 10:59:10.990401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:11.049194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.080351] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.140184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:11.143169] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:11.143586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.163509] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:11.164122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:11.168470] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:11.168508] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:11.184526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:11.217916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.236641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:11.240358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.299503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:11.353503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.531437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=1034] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:11.531510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:11.573894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:11.943847] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:11.943885] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667758, retry interval=1000000, ret=-4179) [2022-08-08 10:59:11.985626] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:11.985664] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667763, retry interval=1000000, ret=-4179) [2022-08-08 10:59:12.170967] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:12.174578] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:12.174606] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:12.684236] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:12.966230] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:12.966269] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667757, retry interval=1000000, ret=-4179) [2022-08-08 10:59:13.033628] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:13.033692] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667762, retry interval=1000000, ret=-4179) [2022-08-08 10:59:13.180293] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:13.185003] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:13.185043] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:13.936959] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927553936951, cur_ts_=1659927553905474, delta=31477) [2022-08-08 10:59:13.988847] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:13.988903] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667756, retry interval=1000000, ret=-4179) [2022-08-08 10:59:14.056429] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:14.056465] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667761, retry interval=1000000, ret=-4179) [2022-08-08 10:59:14.187664] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:14.191021] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:14.191111] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:14.344397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:14.451112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:14.477581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.498733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.662557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.667288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.674257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:14.688021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:14.696014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:14.697854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.719038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:14.796618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.806844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:14.837471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:14.838135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:14.838200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.838880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.845822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:14.890127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.892469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:14.897610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:14.952552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.952667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.961884] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:14.963067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:14.964280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:14.964832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.967906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=330] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.973361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:14.983772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:14.984829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:14.993443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:14.994444] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.008866] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:15.008932] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667755, retry interval=1000000, ret=-4179) [2022-08-08 10:59:15.020539] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:15.026000] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:15.030324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:15.030845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:15.031818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:15.077998] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:15.078049] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667760, retry interval=1000000, ret=-4179) [2022-08-08 10:59:15.078744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:15.083422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:15.114203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.119735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.126038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:15.140474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:15.177112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:15.181010] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927555181002, cur_ts_=1659927555151175, delta=29827) [2022-08-08 10:59:15.187323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:15.194013] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:15.197857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=478] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.200098] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:15.200135] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:15.285653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.286078] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:15.296746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.307903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.345898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:15.376297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.470401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:15.542848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.762108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:15.886107] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:59:16.028608] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:16.028834] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=178] [dc=0] task process failed, start retry(max retry time=9223372036854667754, retry interval=1000000, ret=-4179) [2022-08-08 10:59:16.103630] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:16.103675] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667759, retry interval=1000000, ret=-4179) [2022-08-08 10:59:16.202887] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:16.206687] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:16.206738] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:17.051626] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:17.051688] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667753, retry interval=1000000, ret=-4179) [2022-08-08 10:59:17.122529] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:17.122580] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667758, retry interval=1000000, ret=-4179) [2022-08-08 10:59:17.129994] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:17.209291] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:17.212415] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:17.212460] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:17.530372] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=46706) [2022-08-08 10:59:17.530438] WARN [STORAGE.TRANS] submit_log_impl_ (ob_trans_part_ctx.cpp:6697) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] submitting log costs too much time(ret=0, ret="OB_SUCCESS", timeguard=time guard 'submit_log_impl' cost too much time, used=46805, time_dist: 1,21,46777,2, trans_id={hash:15938641717713447850, inc:84098936, addr:"192.168.11.128:2882", t:1659927557482821}, partition={tid:1099511628002, partition_id:0, part_cnt:0}, state={prepare_version:1659927557482821, state:0}, real_log_type=64, log_size=464) [2022-08-08 10:59:17.530449] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard 'submit_log_impl' cost too much time, used=46829, time_dist: 1,21,46777,2) [2022-08-08 10:59:17.555762] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10615) [2022-08-08 10:59:17.557687] TRACE [TRACE]ob_trans_ctx.cpp:487 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=11] [dc=0] [trans warn] (TRACE=begin_ts=1659927557483442 2022-08-08 02:59:17.483442|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:15938641717713447850, inc:84098936, addr:{ip:"192.168.11.128", port:2882}, t:1659927557482821}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC08F-0-0, start_time:1659927557483326, snapshot:1659927557449350|[end_task] u=152 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927557483613|[calc_checksum_by_commit] u=19 ret:0, checksum:1419500540, checksum_log_ts:9223372036854775807|[submit_log] u=46782 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=38 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=26022 trans_version:1659927557482821, uref:1073741825|[exiting] u=48 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=4 ret:0, log_type:64, log_id:8316192, logging:0, pending:0, uref:1|[end_trans_cb] u=1096 ret:0, param:0, param:true|[destroy] u=9 arg1:140085233547952|total_timeu=74193) [2022-08-08 10:59:17.557784] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=74833, time_dist: generate_cluster_version=74827) [2022-08-08 10:59:17.602025] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927557602016, cur_ts_=1659927557581758, delta=20258) [2022-08-08 10:59:17.610289] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32216) [2022-08-08 10:59:17.704162] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4216] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=88520) [2022-08-08 10:59:17.768356] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=878] [dc=0] clock out of order(cur_ts=1659927557768351, cur_ts_=1659927557735030, delta=33321) [2022-08-08 10:59:17.778112] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=73890) [2022-08-08 10:59:17.778197] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=16] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:7822881127400218032, inc:84098943, addr:"192.168.11.128:2882", t:1659927557615809}, log_id=8301597, used_time=127759) [2022-08-08 10:59:17.778307] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:3168078693282228033, inc:84098946, addr:"192.168.11.128:2882", t:1659927557672095}, log_id=8316193, used_time=105167) [2022-08-08 10:59:17.778420] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=105962, time_dist: do_cluster_heartbeat=8, generate_cluster_version=105950, cluster_service_self_check=3) [2022-08-08 10:59:17.778575] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=129153, time_dist: do_cluster_heartbeat=7, generate_cluster_version=129145) [2022-08-08 10:59:17.855315] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10911) [2022-08-08 10:59:17.855801] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17079, time_dist: do_cluster_heartbeat=12, generate_cluster_version=17064) [2022-08-08 10:59:17.905814] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927557905806, cur_ts_=1659927557884896, delta=20910) [2022-08-08 10:59:17.922868] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19287, time_dist: do_cluster_heartbeat=11, generate_cluster_version=19272) [2022-08-08 10:59:17.936587] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14199) [2022-08-08 10:59:17.938308] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21429, time_dist: print_stat=10, generate_cluster_version=21414) [2022-08-08 10:59:17.993952] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:18.034406] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29874) [2022-08-08 10:59:18.086569] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927558086563, cur_ts_=1659927558049855, delta=36708) [2022-08-08 10:59:18.110722] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:18.110781] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667752, retry interval=1000000, ret=-4179) [2022-08-08 10:59:18.186321] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:18.186378] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667757, retry interval=1000000, ret=-4179) [2022-08-08 10:59:18.188006] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11323) [2022-08-08 10:59:18.193007] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24227, time_dist: generate_cluster_version=24223) [2022-08-08 10:59:18.198169] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28125, time_dist: generate_cluster_version=28120) [2022-08-08 10:59:18.214717] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:18.218892] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:18.218932] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:18.305066] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3254] [dc=0] clock out of order(cur_ts=1659927558305060, cur_ts_=1659927558281180, delta=23880) [2022-08-08 10:59:18.456095] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927558456088, cur_ts_=1659927558430447, delta=25641) [2022-08-08 10:59:18.557039] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22627) [2022-08-08 10:59:18.557246] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23611, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23598) [2022-08-08 10:59:18.586297] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927558586286, cur_ts_=1659927558560902, delta=25384) [2022-08-08 10:59:18.601513] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=44429) [2022-08-08 10:59:18.604132] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=56956, time_dist: do_cluster_heartbeat=8, generate_cluster_version=56943) [2022-08-08 10:59:18.722258] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927558722251, cur_ts_=1659927558682049, delta=40202) [2022-08-08 10:59:18.773923] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=55675, time_dist: do_cluster_heartbeat=7, generate_cluster_version=55663) [2022-08-08 10:59:18.822508] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927558822499, cur_ts_=1659927558769833, delta=52666) [2022-08-08 10:59:18.874272] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=104416) [2022-08-08 10:59:18.874307] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] slow flush(flush_time=14, io_time=104443, file_id=237, flush_start_offset=51750888, file_offset=51751509) [2022-08-08 10:59:18.875101] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:3757958455418647129, inc:84098998, addr:"192.168.11.128:2882", t:1659927558682049}, log_id=8301602, used_time=153906) [2022-08-08 10:59:18.875300] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=155190, time_dist: do_cluster_heartbeat=10, generate_cluster_version=155178) [2022-08-08 10:59:18.927633] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927558927563, cur_ts_=1659927558769833, delta=157730) [2022-08-08 10:59:19.059543] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=476] [dc=0] clock out of order(cur_ts=1659927559059522, cur_ts_=1659927558927563, delta=131959) [2022-08-08 10:59:19.180713] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927559180703, cur_ts_=1659927559063047, delta=117656) [2022-08-08 10:59:19.182770] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=194470) [2022-08-08 10:59:19.185241] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=4] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:11604312687203851545, inc:84099004, addr:"192.168.11.128:2882", t:1659927558927563}, log_id=8301603, used_time=197164) [2022-08-08 10:59:19.185384] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:19.185427] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667751, retry interval=1000000, ret=-4179) [2022-08-08 10:59:19.233002] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:19.297587] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927559297578, cur_ts_=1659927559181786, delta=115792) [2022-08-08 10:59:19.316599] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=133773, io_time=194502, file_id=237, flush_start_offset=51751509, file_offset=51752131) [2022-08-08 10:59:19.363877] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=376833, time_dist: do_cluster_heartbeat=37, generate_cluster_version=376792) [2022-08-08 10:59:19.497609] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927559497599, cur_ts_=1659927559181786, delta=315813) [2022-08-08 10:59:19.512825] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:19.512863] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:19.583555] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=266924) [2022-08-08 10:59:19.583588] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] slow flush(flush_time=12, io_time=266948, file_id=237, flush_start_offset=51752131, file_offset=51752747) [2022-08-08 10:59:19.583630] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:289898277788798639, inc:84099005, addr:"192.168.11.128:2882", t:1659927558927563}, log_id=8316200, used_time=575519) [2022-08-08 10:59:19.583751] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=576916, time_dist: do_cluster_heartbeat=11, generate_cluster_version=576903) [2022-08-08 10:59:19.607472] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927559607462, cur_ts_=1659927559543192, delta=64270) [2022-08-08 10:59:19.652383] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:19.652612] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667756, retry interval=1000000, ret=-4179) [2022-08-08 10:59:19.669073] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=85466) [2022-08-08 10:59:19.669152] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] transaction log sync use too much time(partition={tid:1099511627916, partition_id:0, part_cnt:0}, trans_id={hash:10530431996842917190, inc:84099010, addr:"192.168.11.128:2882", t:1659927559063047}, log_id=988015, used_time=482873) [2022-08-08 10:59:19.669226] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:3890600336733391732, inc:84099015, addr:"192.168.11.128:2882", t:1659927559181786}, log_id=8301604, used_time=239657) [2022-08-08 10:59:19.692746] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=266905, time_dist: print_stat=11, do_cluster_heartbeat=3, generate_cluster_version=266887) [2022-08-08 10:59:19.715791] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927559715784, cur_ts_=1659927559695783, delta=20001) [2022-08-08 10:59:19.780244] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:59:19.780271] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:59:19.853532] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE3092-0-0] [lt=18] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927559852225 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927559785033 abs_expired_time=1659927561782383 stmt_timeout=1997350 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21593503 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927559714344 start_version=-1 trans_version=1659927559777064 commit_version=0 stmt_start_time=1659927559695783 abs_expired_time=1659927589653895 stmt_timeout=29958112 abs_lock_wait_timeout=1659927589653895 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:7626438918355306746, inc:84099023, addr:"192.168.11.128:2882", t:1659927559695783} checksum_log_ts=9223372036854775807, version=1659927559852225) [2022-08-08 10:59:19.853959] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927559853954, cur_ts_=1659927559785033, delta=68921) [2022-08-08 10:59:19.919963] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=249946) [2022-08-08 10:59:19.920027] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] slow flush(flush_time=15, io_time=250002, file_id=237, flush_start_offset=51754008, file_offset=51754695) [2022-08-08 10:59:19.920085] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=14] [dc=0] transaction log sync use too much time(partition={tid:1099511627916, partition_id:0, part_cnt:0}, trans_id={hash:73088450363090678, inc:84099022, addr:"192.168.11.128:2882", t:1659927559607462}, log_id=988016, used_time=250147) [2022-08-08 10:59:19.932022] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=5] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511627916, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:988016, proposal_id:{time_to_usec:1659815620310360, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:51754056, size:639}, after_consume_timestamp:1659927559785033}, end_time=1659927559931753, begin_time=1659927559785033, cost_time=146720) [2022-08-08 10:59:20.048360] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] clock out of order(cur_ts=1659927560048348, cur_ts_=1659927560027908, delta=20440) [2022-08-08 10:59:20.065795] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=145743) [2022-08-08 10:59:20.065841] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] slow flush(flush_time=12, io_time=145784, file_id=237, flush_start_offset=51754695, file_offset=51757934) [2022-08-08 10:59:20.105197] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=13] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:7626438918355306746, inc:84099023, addr:"192.168.11.128:2882", t:1659927559695783}, log_id=1655770, used_time=322250) [2022-08-08 10:59:20.105298] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=15] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:5090249626088052295, inc:84099025, addr:"192.168.11.128:2882", t:1659927559777064}, log_id=8316201, used_time=321550) [2022-08-08 10:59:20.105349] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=5] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:6362121510932380347, inc:84099028, addr:"192.168.11.128:2882", t:1659927559783777}, log_id=8301605, used_time=320273) [2022-08-08 10:59:20.105521] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=322479, time_dist: do_cluster_heartbeat=6, generate_cluster_version=322471) [2022-08-08 10:59:20.105578] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=321099, time_dist: do_cluster_heartbeat=6, generate_cluster_version=321092) [2022-08-08 10:59:20.234786] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927560234773, cur_ts_=1659927560214480, delta=20293) [2022-08-08 10:59:20.281356] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:20.281386] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=174325) [2022-08-08 10:59:20.281394] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667750, retry interval=1000000, ret=-4179) [2022-08-08 10:59:20.281410] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] slow flush(flush_time=12, io_time=174339, file_id=237, flush_start_offset=51757934, file_offset=51758724) [2022-08-08 10:59:20.281458] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=9] [dc=0] transaction log sync use too much time(partition={tid:1099511627777, partition_id:0, part_cnt:1}, trans_id={hash:1781425704455870458, inc:84099032, addr:"192.168.11.128:2882", t:1659927560062224}, log_id=1655771, used_time=174442) [2022-08-08 10:59:20.341226] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=59796) [2022-08-08 10:59:20.345458] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927560345445, cur_ts_=1659927560300521, delta=44924) [2022-08-08 10:59:20.356173] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=29] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:12713993600352808644, inc:84099035, addr:"192.168.11.128:2882", t:1659927560185237}, log_id=8301606, used_time=166630) [2022-08-08 10:59:20.361593] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=31] [dc=0] flush_cb cost too much time(partition_key_={tid:1100611139404002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8301606, proposal_id:{time_to_usec:1659815624166414, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:51758772, size:574}, after_consume_timestamp:1659927560345445}, end_time=1659927560360749, begin_time=1659927560345445, cost_time=15304) [2022-08-08 10:59:20.361628] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=17] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:6069469831658009371, inc:84099036, addr:"192.168.11.128:2882", t:1659927560185237}, log_id=8316202, used_time=171541) [2022-08-08 10:59:20.370157] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=180589, time_dist: do_cluster_heartbeat=4, generate_cluster_version=180568) [2022-08-08 10:59:20.375955] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=187077, time_dist: do_cluster_heartbeat=13, generate_cluster_version=187059) [2022-08-08 10:59:20.395772] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=54482) [2022-08-08 10:59:20.401229] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=3] [dc=0] transaction log sync use too much time(partition={tid:1099511627916, partition_id:0, part_cnt:0}, trans_id={hash:3374764725085638212, inc:84099041, addr:"192.168.11.128:2882", t:1659927560282274}, log_id=988017, used_time=118215) [2022-08-08 10:59:20.463095] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927560463083, cur_ts_=1659927560421940, delta=41143) [2022-08-08 10:59:20.503662] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=40470) [2022-08-08 10:59:20.504494] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=82033, time_dist: do_cluster_heartbeat=8, generate_cluster_version=82021) [2022-08-08 10:59:20.549431] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=45706) [2022-08-08 10:59:20.549694] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=86149, time_dist: do_cluster_heartbeat=12, generate_cluster_version=86135) [2022-08-08 10:59:20.574683] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12350) [2022-08-08 10:59:20.579776] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20227, time_dist: do_cluster_heartbeat=9, generate_cluster_version=20214) [2022-08-08 10:59:20.608829] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:20.618339] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:20.618407] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:20.625878] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13636, time_dist: print_stat=9, do_cluster_heartbeat=3, generate_cluster_version=13618) [2022-08-08 10:59:20.691832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:20.692723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:20.696097] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:20.696135] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667755, retry interval=1000000, ret=-4179) [2022-08-08 10:59:20.701099] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23946) [2022-08-08 10:59:20.702398] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25933, time_dist: do_cluster_heartbeat=3, generate_cluster_version=25925) [2022-08-08 10:59:20.826321] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17516) [2022-08-08 10:59:20.826578] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18438, time_dist: generate_cluster_version=18436) [2022-08-08 10:59:20.894799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:21.006915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:21.038914] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39834) [2022-08-08 10:59:21.039150] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=41016, time_dist: generate_cluster_version=41014) [2022-08-08 10:59:21.071070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.071913] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927561071907, cur_ts_=1659927561042379, delta=29528) [2022-08-08 10:59:21.105821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.159494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:21.159899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.164318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:21.181117] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:21.202081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:21.239039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.251777] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:21.257102] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14868384}) [2022-08-08 10:59:21.275838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.307434] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:21.308175] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=671] [dc=0] task process failed, start retry(max retry time=9223372036854667749, retry interval=1000000, ret=-4179) [2022-08-08 10:59:21.321044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:21.369537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.471855] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927558337402, end_time=1659927561471841, elapsed_time=3134439, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:59:21.547637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:21.547672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:21.591484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:21.622447] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:21.625503] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:21.625583] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:21.715378] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:21.715440] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667754, retry interval=1000000, ret=-4179) [2022-08-08 10:59:22.012329] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927562012321, cur_ts_=1659927561990643, delta=21678) [2022-08-08 10:59:22.327459] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:22.327492] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667748, retry interval=1000000, ret=-4179) [2022-08-08 10:59:22.594845] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927562593760 2022-08-08 02:59:22.593760|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:314405338772341594, inc:84099121, addr:{ip:"192.168.11.128", port:2882}, t:1659927562592000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741826, trace_id:YB42C0A80B80-0005E3CFCB9A62D5-0-0, start_time:1659927562593634, snapshot:1659927562582211|[end_task] u=129 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927562593907|[calc_checksum_by_commit] u=21 ret:0, checksum:744678821, checksum_log_ts:9223372036854775807|[submit_log] u=62 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=791 trans_version:1659927562593930, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301619, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=1047) [2022-08-08 10:59:22.628151] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:22.631264] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:22.631302] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:22.739355] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:22.739462] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] task process failed, start retry(max retry time=9223372036854667753, retry interval=1000000, ret=-4179) [2022-08-08 10:59:23.346788] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:23.346829] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667747, retry interval=1000000, ret=-4179) [2022-08-08 10:59:23.574239] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927563574233, cur_ts_=1659927563549104, delta=25129) [2022-08-08 10:59:23.597807] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12515) [2022-08-08 10:59:23.598036] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13581, time_dist: do_cluster_heartbeat=8, generate_cluster_version=13570) [2022-08-08 10:59:23.650937] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:23.656868] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:23.656914] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:23.780644] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:23.780712] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667752, retry interval=1000000, ret=-4179) [2022-08-08 10:59:24.365345] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:24.365398] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667746, retry interval=1000000, ret=-4179) [2022-08-08 10:59:24.368897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:24.474105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:24.496775] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:24.514747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.659562] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:24.662843] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:24.662944] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:24.687295] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:24.690703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:24.690819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:24.708198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:24.718594] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=623] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:24.720478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:24.743385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:24.805691] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:24.805856] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667751, retry interval=1000000, ret=-4179) [2022-08-08 10:59:24.812958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=191] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.824622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:24.853459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:24.853495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:24.855942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.860488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.863897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:24.911586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.911829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:24.918118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:24.970628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:24.977929] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:24.978148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:24.978243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:24.981935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:24.988530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.988720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.998968] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=192] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:24.999129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:25.009369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:25.010670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.019694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.040426] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:25.044432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:25.050954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:25.051292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:25.054772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:25.102307] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:25.103662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:25.138197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.138485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.151489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:25.161034] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:25.200224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:25.211239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:25.221695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=223] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.304843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.305838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:25.317228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.326747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.372015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:25.391085] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:25.391135] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667745, retry interval=1000000, ret=-4179) [2022-08-08 10:59:25.395774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.495791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:25.562282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.668098] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:25.671330] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:25.671364] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:25.781492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:25.826960] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:25.827002] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667750, retry interval=1000000, ret=-4179) [2022-08-08 10:59:25.891634] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927565891628, cur_ts_=1659927565855111, delta=36517) [2022-08-08 10:59:26.030045] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16308) [2022-08-08 10:59:26.030265] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17292, time_dist: do_cluster_heartbeat=9, generate_cluster_version=17281) [2022-08-08 10:59:26.412035] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:26.412073] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667744, retry interval=1000000, ret=-4179) [2022-08-08 10:59:26.603783] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] clock out of order(cur_ts=1659927566603772, cur_ts_=1659927566566815, delta=36957) [2022-08-08 10:59:26.673945] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:26.677257] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:26.677280] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:26.846942] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:26.847051] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667749, retry interval=1000000, ret=-4179) [2022-08-08 10:59:26.908215] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927566908207, cur_ts_=1659927566887749, delta=20458) [2022-08-08 10:59:27.156221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:27.435961] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:27.436115] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] task process failed, start retry(max retry time=9223372036854667743, retry interval=1000000, ret=-4179) [2022-08-08 10:59:27.681468] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:27.684926] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:27.684959] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:27.864526] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12159) [2022-08-08 10:59:27.865849] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:27.865887] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667748, retry interval=1000000, ret=-4179) [2022-08-08 10:59:27.866408] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16379, time_dist: generate_cluster_version=16376) [2022-08-08 10:59:28.097196] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:28.457363] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:28.457404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667742, retry interval=1000000, ret=-4179) [2022-08-08 10:59:28.687261] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:28.690903] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:28.690940] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:28.909711] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927568909707, cur_ts_=1659927568889122, delta=20585) [2022-08-08 10:59:28.912461] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:28.912533] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667747, retry interval=1000000, ret=-4179) [2022-08-08 10:59:29.047617] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12159) [2022-08-08 10:59:29.048092] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20736, time_dist: do_cluster_heartbeat=7, generate_cluster_version=20726) [2022-08-08 10:59:29.048515] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21997, time_dist: do_cluster_heartbeat=10, generate_cluster_version=21985) [2022-08-08 10:59:29.481132] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:29.481321] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:29.481932] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:29.482917] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=924] [dc=0] task process failed, start retry(max retry time=9223372036854667741, retry interval=1000000, ret=-4179) [2022-08-08 10:59:29.694678] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:29.701688] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:29.701714] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:29.772055] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:59:29.772323] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=169] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:59:29.940443] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:29.940546] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:29.942140] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:29.942190] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667746, retry interval=1000000, ret=-4179) [2022-08-08 10:59:29.987361] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14872480}) [2022-08-08 10:59:30.044075] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659927570044068, cur_ts_=1659927570011767, delta=32301) [2022-08-08 10:59:30.207162] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927568338033, end_time=1659927570207150, elapsed_time=1869117, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:59:30.517613] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:30.517756] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:30.519123] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:30.519187] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667740, retry interval=1000000, ret=-4179) [2022-08-08 10:59:30.703491] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:30.706782] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:30.706849] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:30.711962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:30.713943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:30.920462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:30.965001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:30.965174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:30.966493] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:30.966543] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667745, retry interval=1000000, ret=-4179) [2022-08-08 10:59:31.026853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:31.087100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.102481] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25361) [2022-08-08 10:59:31.102696] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26170, time_dist: generate_cluster_version=26168) [2022-08-08 10:59:31.125907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.184110] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:31.185665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.186301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:31.197671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:31.198045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30B3-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.198100] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30B3-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.198118] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30B3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.198125] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30B3-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.198223] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.198275] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.200120] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:31.221291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:31.274196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.275582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:31.285394] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14127, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14113) [2022-08-08 10:59:31.286044] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12253, time_dist: generate_cluster_version=12250) [2022-08-08 10:59:31.305550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.338945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:31.391459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.532889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:31.532916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:31.551432] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:31.551496] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667739, retry interval=1000000, ret=-4179) [2022-08-08 10:59:31.571676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:31.572333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:31.610827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:31.709389] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:31.713736] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:31.713835] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:31.969698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49ED-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:31.970009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49ED-0-0] [lt=186] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:31.970050] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49ED-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:31.995509] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:31.995603] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] task process failed, start retry(max retry time=9223372036854667744, retry interval=1000000, ret=-4179) [2022-08-08 10:59:32.209102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:32.209220] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:32.576550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:32.576757] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:32.578157] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:32.578195] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667738, retry interval=1000000, ret=-4179) [2022-08-08 10:59:32.684550] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:32.716285] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:32.720509] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:32.720547] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:33.020372] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:33.020474] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667743, retry interval=1000000, ret=-4179) [2022-08-08 10:59:33.619627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:33.620427] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:33.621999] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:33.622035] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667737, retry interval=1000000, ret=-4179) [2022-08-08 10:59:33.723059] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:33.726823] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:33.726933] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:33.828953] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11678, time_dist: generate_cluster_version=11673) [2022-08-08 10:59:34.042401] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:34.042453] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667742, retry interval=1000000, ret=-4179) [2022-08-08 10:59:34.222389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.222429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.222497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.222543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.222694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.222899] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.223039] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.224334] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.224473] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.224582] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.224897] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30BC-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.225026] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.225142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.225658] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.225842] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:34.392125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:34.497596] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:34.521855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:34.531457] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:34.644240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:34.644433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:34.645709] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:34.645758] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667736, retry interval=1000000, ret=-4179) [2022-08-08 10:59:34.648270] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] [trans sampling] (TRACE=begin_ts=1659927574646257 2022-08-08 02:59:34.646257|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:306225802479352118, inc:84099588, addr:{ip:"192.168.11.128", port:2882}, t:1659927574645000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7DC4-0-0, start_time:1659927574646132, snapshot:1659927574611247|[end_task] u=405 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927574646682|[calc_checksum_by_commit] u=36 ret:0, checksum:1306768777, checksum_log_ts:9223372036854775807|[submit_log] u=44 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1422 trans_version:1659927574645336, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:988045, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=1967) [2022-08-08 10:59:34.707382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:34.707898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:34.710947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:34.729858] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:34.733161] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:34.733197] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:34.735355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:34.737145] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:34.739182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:34.760065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:34.836880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:34.842529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:34.872577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:34.875584] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:34.875670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:34.883115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:34.885981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:34.931115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:34.935106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:34.941577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:34.993555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:34.993708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:34.994296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:34.994834] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:35.003529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:35.004142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.015556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.030795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.036234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.036774] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927575036768, cur_ts_=1659927575010990, delta=25778) [2022-08-08 10:59:35.036907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.037137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:35.053734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:35.057417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:35.069514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:35.069698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.069833] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15445) [2022-08-08 10:59:35.072925] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23886, time_dist: do_cluster_heartbeat=132, generate_cluster_version=23749) [2022-08-08 10:59:35.075008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.091835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.108754] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:35.108797] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667741, retry interval=1000000, ret=-4179) [2022-08-08 10:59:35.123063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:35.124109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:35.153784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.155298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.176216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:35.186138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:35.224214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:35.228811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:35.237893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.325719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.325852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:35.340795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.351278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.388889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:35.434069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.434882] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927575434876, cur_ts_=1659927575409900, delta=24976) [2022-08-08 10:59:35.517838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:35.585821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.671301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:35.671434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:35.673071] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:35.673125] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667735, retry interval=1000000, ret=-4179) [2022-08-08 10:59:35.738634] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] clock out of order(cur_ts=1659927575738625, cur_ts_=1659927575714440, delta=24185) [2022-08-08 10:59:35.739754] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:35.745714] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:35.745737] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:35.806095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:35.909384] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:59:36.132946] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.133411] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=443] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.134788] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:36.134857] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667740, retry interval=1000000, ret=-4179) [2022-08-08 10:59:36.243999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30C2-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:36.244104] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30C2-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:36.244358] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30C2-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:36.244641] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:36.244784] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:36.244933] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:36.366415] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927576366408, cur_ts_=1659927576333851, delta=32557) [2022-08-08 10:59:36.402697] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13764) [2022-08-08 10:59:36.427650] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13880) [2022-08-08 10:59:36.684006] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61BF-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.684116] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61BF-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.686139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.686297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.704218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:36.705410] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:36.705471] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667734, retry interval=1000000, ret=-4179) [2022-08-08 10:59:36.748253] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:36.751451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:36.751490] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:37.140255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.155969] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.161820] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:37.161907] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667739, retry interval=1000000, ret=-4179) [2022-08-08 10:59:37.181473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:37.708048] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49F6-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.708104] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49F6-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.708121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49F6-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.710171] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.710214] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:37.736667] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:37.736718] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667733, retry interval=1000000, ret=-4179) [2022-08-08 10:59:37.753694] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:37.756857] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:37.756922] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:38.161752] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:38.186923] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:38.186961] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667738, retry interval=1000000, ret=-4179) [2022-08-08 10:59:38.509929] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927578509922, cur_ts_=1659927578488857, delta=21065) [2022-08-08 10:59:38.659191] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927578659185, cur_ts_=1659927578635737, delta=23448) [2022-08-08 10:59:38.714949] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=45357) [2022-08-08 10:59:38.715205] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=46259, time_dist: do_cluster_heartbeat=6, generate_cluster_version=46251) [2022-08-08 10:59:38.735957] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18726, time_dist: generate_cluster_version=18723) [2022-08-08 10:59:38.760530] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:38.764091] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:38.764128] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:38.770713] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:38.770765] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667732, retry interval=1000000, ret=-4179) [2022-08-08 10:59:39.210079] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:39.210113] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667737, retry interval=1000000, ret=-4179) [2022-08-08 10:59:39.268129] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30CB-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.268168] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30CB-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.268518] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30CB-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.268535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30CB-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.268859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.269008] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.269274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.269467] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=173] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:39.766425] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:39.769624] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:39.769686] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:39.779032] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=817] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:59:39.779057] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:59:39.799267] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:39.799347] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:39.800548] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:39.800625] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667731, retry interval=1000000, ret=-4179) [2022-08-08 10:59:39.981842] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14876864}) [2022-08-08 10:59:40.037949] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] clock out of order(cur_ts=1659927580037943, cur_ts_=1659927580017420, delta=20523) [2022-08-08 10:59:40.174872] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927578338948, end_time=1659927580174858, elapsed_time=1835910, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:59:40.213011] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C49F9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:40.233257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:40.234790] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:40.234853] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667736, retry interval=1000000, ret=-4179) [2022-08-08 10:59:40.280406] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:40.280471] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:40.280780] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:40.675916] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=5] [dc=0] [trans sampling] (TRACE=begin_ts=1659927580675167 2022-08-08 02:59:40.675167|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:17578870387110741986, inc:84099814, addr:{ip:"192.168.11.128", port:2882}, t:1659927580674000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC100-0-0, start_time:1659927580675040, snapshot:1659927580639997|[end_task] u=126 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927580675309|[calc_checksum_by_commit] u=18 ret:0, checksum:1682027853, checksum_log_ts:9223372036854775807|[submit_log] u=53 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=461 trans_version:1659927580674137, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8316305, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=710) [2022-08-08 10:59:40.732222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:40.737160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:40.772451] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:40.776690] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=844] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:40.776744] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:40.825134] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:40.825166] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667730, retry interval=1000000, ret=-4179) [2022-08-08 10:59:40.937275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:41.045239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:41.107772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.149121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.188663] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927581188656, cur_ts_=1659927581165578, delta=23078) [2022-08-08 10:59:41.208503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:41.209052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.209632] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:41.218960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:41.243783] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:41.265516] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:41.265600] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] task process failed, start retry(max retry time=9223372036854667735, retry interval=1000000, ret=-4179) [2022-08-08 10:59:41.295183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:41.296656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.328526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=2212] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.358149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:41.410255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.586546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=670] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:41.589485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:41.627618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:41.660919] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927581660910, cur_ts_=1659927581638457, delta=22453) [2022-08-08 10:59:41.779323] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:41.784532] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:41.784561] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:41.849342] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:41.852075] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:41.852118] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667729, retry interval=1000000, ret=-4179) [2022-08-08 10:59:42.108797] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927582108792, cur_ts_=1659927582083738, delta=25054) [2022-08-08 10:59:42.268141] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49BF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:42.268286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49BF-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:42.287769] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:42.288003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:42.292722] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:42.292766] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667734, retry interval=1000000, ret=-4179) [2022-08-08 10:59:42.787638] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:42.792944] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:42.793021] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:42.879482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:42.879659] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:42.881125] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:42.881177] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667728, retry interval=1000000, ret=-4179) [2022-08-08 10:59:43.210500] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927583210492, cur_ts_=1659927583178179, delta=32313) [2022-08-08 10:59:43.335696] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927583335690, cur_ts_=1659927583309199, delta=26491) [2022-08-08 10:59:43.384336] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=56072) [2022-08-08 10:59:43.391745] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:43.391783] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667733, retry interval=1000000, ret=-4179) [2022-08-08 10:59:43.401069] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=73802, time_dist: print_stat=9, do_cluster_heartbeat=3, generate_cluster_version=73785) [2022-08-08 10:59:43.402874] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18492) [2022-08-08 10:59:43.403281] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=71222, time_dist: do_cluster_heartbeat=33, generate_cluster_version=71185) [2022-08-08 10:59:43.545328] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927583545322, cur_ts_=1659927583525314, delta=20008) [2022-08-08 10:59:43.676215] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13089, time_dist: do_cluster_heartbeat=11, generate_cluster_version=13073) [2022-08-08 10:59:43.737741] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6657] [dc=0] clock out of order(cur_ts=1659927583737732, cur_ts_=1659927583713089, delta=24643) [2022-08-08 10:59:43.763477] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33802) [2022-08-08 10:59:43.766737] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=25] [dc=0] [trans sampling] (TRACE=begin_ts=1659927583708230 2022-08-08 02:59:43.708230|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:17305037828401886855, inc:84099928, addr:{ip:"192.168.11.128", port:2882}, t:1659927583708000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC10F-0-0, start_time:1659927583708078, snapshot:1659927583684938|[end_task] u=133 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927583708380|[calc_checksum_by_commit] u=18 ret:0, checksum:3838523456, checksum_log_ts:9223372036854775807|[submit_log] u=29 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=58170 trans_version:1659927583708132, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8316320, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=58398) [2022-08-08 10:59:43.766869] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=59123, time_dist: do_cluster_heartbeat=8, generate_cluster_version=59114) [2022-08-08 10:59:43.805955] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:43.810935] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:43.810975] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:43.895221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:43.895272] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:43.918543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:43.918570] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:43.925054] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=314] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:43.925102] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667727, retry interval=1000000, ret=-4179) [2022-08-08 10:59:44.410086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:44.417519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:44.417544] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:44.418330] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:44.418377] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667732, retry interval=1000000, ret=-4179) [2022-08-08 10:59:44.522597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:44.544038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:44.556835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:44.726654] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:44.729497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:44.732946] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:44.753902] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:44.760763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:44.764219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:44.781550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:44.829735] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:44.833258] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:44.833320] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:44.857695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:44.858468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:44.889719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:44.896785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:44.900083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:44.906146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:44.907135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:44.952947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:44.953195] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:44.954430] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:44.954791] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:44.954847] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667726, retry interval=1000000, ret=-4179) [2022-08-08 10:59:44.958704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:44.958732] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.016774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:45.016829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:45.016840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:45.016971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:45.027230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.033289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:45.037613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.050456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.058482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=18659] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:45.059462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.060797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.075162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=2024] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:45.080151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=299] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:45.085441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.093030] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.093089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:45.114416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.145585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:45.147125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:45.172392] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.172931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.198166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:45.208185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:45.246831] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:45.246889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:45.259708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.342107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:45.349421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.359528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.375084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.408727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:45.465431] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=220] [dc=0] clock out of order(cur_ts=1659927585465421, cur_ts_=1659927585426330, delta=39091) [2022-08-08 10:59:45.465860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.490801] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:45.490846] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:45.492150] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:45.492198] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667731, retry interval=1000000, ret=-4179) [2022-08-08 10:59:45.535298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:45.605015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.824638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:45.836791] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:45.840807] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:45.840861] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:45.975323] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=3040] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13488) [2022-08-08 10:59:45.978377] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17483, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17469) [2022-08-08 10:59:45.981948] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:45.981968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:45.982006] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:45.983433] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:45.983469] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667725, retry interval=1000000, ret=-4179) [2022-08-08 10:59:46.484271] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:46.484377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:46.484661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:46.484820] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:46.500694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:46.500743] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:46.550810] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:46.550962] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:46.557847] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:46.557918] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667730, retry interval=1000000, ret=-4179) [2022-08-08 10:59:46.746153] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659927586745236 2022-08-08 02:59:46.745236|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:9063351837424792533, inc:84100037, addr:{ip:"192.168.11.128", port:2882}, t:1659927586744000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A634D-0-0, start_time:1659927586745095, snapshot:1659927586699865|[end_task] u=197 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927586745452|[calc_checksum_by_commit] u=21 ret:0, checksum:3639266685, checksum_log_ts:9223372036854775807|[submit_log] u=50 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=550 trans_version:1659927586745476, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301739, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=876) [2022-08-08 10:59:46.826841] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] clock out of order(cur_ts=1659927586826832, cur_ts_=1659927586794954, delta=31878) [2022-08-08 10:59:46.843730] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:46.848228] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:46.848567] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=323] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:46.931109] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927586931101, cur_ts_=1659927586900216, delta=30885) [2022-08-08 10:59:47.047976] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:47.048165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:47.050686] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:47.050739] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667724, retry interval=1000000, ret=-4179) [2022-08-08 10:59:47.201100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:47.513248] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E3-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:47.513275] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E3-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:47.513287] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:47.513295] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E3-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:47.585953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:47.586540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:47.598120] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:47.598189] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667729, retry interval=1000000, ret=-4179) [2022-08-08 10:59:47.602994] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22366) [2022-08-08 10:59:47.605428] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25577, time_dist: generate_cluster_version=25572) [2022-08-08 10:59:47.695963] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927587695958, cur_ts_=1659927587673830, delta=22128) [2022-08-08 10:59:47.862281] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:47.865513] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:47.865549] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:48.057529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:48.057665] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:48.077392] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:48.077664] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] task process failed, start retry(max retry time=9223372036854667723, retry interval=1000000, ret=-4179) [2022-08-08 10:59:48.224188] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:48.519614] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE30E6-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927588518464 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927588518454 abs_expired_time=1659927590448968 stmt_timeout=1930514 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21593906 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927588444236 start_version=-1 trans_version=1659927588518454 commit_version=0 stmt_start_time=1659927588511487 abs_expired_time=1659927618411772 stmt_timeout=29900285 abs_lock_wait_timeout=1659927618411772 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:6916239568468391818, inc:84100109, addr:"192.168.11.128:2882", t:1659927588511487} checksum_log_ts=9223372036854775807, version=1659927588518464) [2022-08-08 10:59:48.659536] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:48.660156] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=586] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:48.673610] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:48.673885] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667728, retry interval=1000000, ret=-4179) [2022-08-08 10:59:48.769805] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927588769796, cur_ts_=1659927588749668, delta=20128) [2022-08-08 10:59:48.847120] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20712) [2022-08-08 10:59:48.848219] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25343, time_dist: generate_cluster_version=25340) [2022-08-08 10:59:48.850861] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25618, time_dist: generate_cluster_version=25315) [2022-08-08 10:59:48.874837] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:48.884111] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:48.884142] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:49.049789] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927589049782, cur_ts_=1659927589028889, delta=20893) [2022-08-08 10:59:49.113297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:49.115646] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:49.115736] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667722, retry interval=1000000, ret=-4179) [2022-08-08 10:59:49.230309] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927589230299, cur_ts_=1659927589209607, delta=20692) [2022-08-08 10:59:49.527209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E9-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:49.527262] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E9-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:49.527281] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E9-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:49.527289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30E9-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:49.527390] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:49.699731] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:49.699771] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667727, retry interval=1000000, ret=-4179) [2022-08-08 10:59:49.770631] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927589763031 2022-08-08 02:59:49.763031|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:6580484613376061088, inc:84100158, addr:{ip:"192.168.11.128", port:2882}, t:1659927589762000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC12D-0-0, start_time:1659927589762878, snapshot:1659927589720704|[end_task] u=141 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927589763190|[calc_checksum_by_commit] u=20 ret:0, checksum:2528808682, checksum_log_ts:9223372036854775807|[submit_log] u=1206 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=6142 trans_version:1659927589762512, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8316350, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=7558) [2022-08-08 10:59:49.774488] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:59:49.774527] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 10:59:49.887519] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:49.891862] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:49.891888] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:50.144141] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:50.151411] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6866] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:50.183452] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:50.183745] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667721, retry interval=1000000, ret=-4179) [2022-08-08 10:59:50.185726] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=486] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11471, time_dist: generate_cluster_version=11467) [2022-08-08 10:59:50.185831] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12373, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12363) [2022-08-08 10:59:50.361033] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14881152}) [2022-08-08 10:59:50.552468] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927588339913, end_time=1659927590552454, elapsed_time=2212541, this=0x7f69b154b010, thread_id=10622) [2022-08-08 10:59:50.723662] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:50.723793] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667726, retry interval=1000000, ret=-4179) [2022-08-08 10:59:50.754116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:50.757537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:50.894381] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:50.899488] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:50.899528] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:50.953922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:51.062667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:51.128131] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.176265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.208052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:51.208089] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:51.209528] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:51.209602] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667720, retry interval=1000000, ret=-4179) [2022-08-08 10:59:51.231232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:51.231310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.239166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=297] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:51.239225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:51.259810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=841] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:51.312181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:51.314539] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.346015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.377505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:51.430906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.542790] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30EF-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:51.602355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:51.605234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:51.656660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:51.656916] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927591656912, cur_ts_=1659927591635059, delta=21853) [2022-08-08 10:59:51.729035] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61D4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:51.729067] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61D4-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:51.749138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:51.749208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:51.750754] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:51.750810] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667725, retry interval=1000000, ret=-4179) [2022-08-08 10:59:51.792518] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10443, time_dist: do_cluster_heartbeat=7, generate_cluster_version=10434) [2022-08-08 10:59:51.921932] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:51.922460] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927591922457, cur_ts_=1659927591890484, delta=31973) [2022-08-08 10:59:51.935234] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:51.935260] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:52.229276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:52.229319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:52.252906] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:52.253024] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667719, retry interval=1000000, ret=-4179) [2022-08-08 10:59:52.307452] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927592307446, cur_ts_=1659927592281817, delta=25629) [2022-08-08 10:59:52.685140] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:52.774851] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:52.776560] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:52.776630] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667724, retry interval=1000000, ret=-4179) [2022-08-08 10:59:52.777851] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=32] [dc=0] [trans sampling] (TRACE=begin_ts=1659927592777050 2022-08-08 02:59:52.777050|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:7774611195165028983, inc:84100271, addr:{ip:"192.168.11.128", port:2882}, t:1659927592776000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7DE7-0-0, start_time:1659927592776943, snapshot:1659927592714364|[end_task] u=93 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927592777159|[calc_checksum_by_commit] u=18 ret:0, checksum:2828501189, checksum_log_ts:9223372036854775807|[submit_log] u=64 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=538 trans_version:1659927592776675, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:988080, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233102640|total_timeu=765) [2022-08-08 10:59:52.939024] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:52.942674] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:52.942711] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:53.256912] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A0D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.260441] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=308] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.260573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.277733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.278976] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:53.279025] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667718, retry interval=1000000, ret=-4179) [2022-08-08 10:59:53.561433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30F5-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:53.561508] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE30F5-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 10:59:53.780296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D1-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.780371] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D1-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.798532] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.798730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:53.800204] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:53.800264] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667723, retry interval=1000000, ret=-4179) [2022-08-08 10:59:53.945388] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:53.950465] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:53.950502] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:54.300355] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:54.300402] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:54.301600] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:54.301660] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667717, retry interval=1000000, ret=-4179) [2022-08-08 10:59:54.430865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:54.543076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:54.563708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:54.574794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:54.743423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:54.752920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:54.754370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:54.823064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:54.823110] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:54.841476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:54.879773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:54.881840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:54.896957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:54.908880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:54.927263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:54.927395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:54.935167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:54.935252] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:54.939673] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927594939668, cur_ts_=1659927594909715, delta=29953) [2022-08-08 10:59:54.954393] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:54.979432] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36602) [2022-08-08 10:59:54.980182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:54.980208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:54.980683] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45204, time_dist: do_cluster_heartbeat=10, generate_cluster_version=45191) [2022-08-08 10:59:54.981289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:54.985429] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:54.987059] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=1608] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:54.988468] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=49712, time_dist: do_cluster_heartbeat=9, generate_cluster_version=49698) [2022-08-08 10:59:55.017504] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:55.017557] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667722, retry interval=1000000, ret=-4179) [2022-08-08 10:59:55.033167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 10:59:55.038177] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:55.040760] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:55.041683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:55.048870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.054799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:55.055192] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.071151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.078297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:55.078301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.083239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.096234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:55.099153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:55.102479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.116859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.116974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:55.133901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.165802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:55.168715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:55.191439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.197185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.220212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:55.231078] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 10:59:55.262099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 10:59:55.271793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:55.282717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.326325] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:55.326456] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:55.327713] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:55.327873] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667716, retry interval=1000000, ret=-4179) [2022-08-08 10:59:55.359508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:55.364589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.377558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.393780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.429137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 10:59:55.488588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.551989] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 10:59:55.626581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.787568] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=21] [dc=0] [trans sampling] (TRACE=begin_ts=1659927595786763 2022-08-08 02:59:55.786763|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:16040942704285893430, inc:84100381, addr:{ip:"192.168.11.128", port:2882}, t:1659927595786000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A637B-0-0, start_time:1659927595786643, snapshot:1659927595743200|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927595786904|[calc_checksum_by_commit] u=20 ret:0, checksum:426892051, checksum_log_ts:9223372036854775807|[submit_log] u=44 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=520 trans_version:1659927595786926, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301785, logging:0, pending:0, uref:1|[end_trans_cb] u=23 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=767) [2022-08-08 10:59:55.850282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 10:59:55.940512] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 10:59:55.991421] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:55.997797] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:55.997901] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:56.041890] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:56.041935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:56.043837] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:56.043890] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667721, retry interval=1000000, ret=-4179) [2022-08-08 10:59:56.339955] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:56.339985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:56.357184] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:56.358756] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:56.358834] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667715, retry interval=1000000, ret=-4179) [2022-08-08 10:59:57.000291] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:57.003597] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:57.003660] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:57.067843] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:57.068018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:57.069359] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:57.069430] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667720, retry interval=1000000, ret=-4179) [2022-08-08 10:59:57.223506] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 10:59:57.449538] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927597449528, cur_ts_=1659927597298781, delta=150747) [2022-08-08 10:59:57.545573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:57.545701] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:57.548447] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:57.548631] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667714, retry interval=1000000, ret=-4179) [2022-08-08 10:59:57.634110] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] clock out of order(cur_ts=1659927597634100, cur_ts_=1659927597603971, delta=30129) [2022-08-08 10:59:57.690859] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10576, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10561) [2022-08-08 10:59:57.698784] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10769, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10754) [2022-08-08 10:59:58.009214] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:58.012459] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:58.012512] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:58.075029] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.075380] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.075464] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.075600] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.075758] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.075881] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.076054] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.076145] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49D7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.079551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.079582] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.098304] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.098354] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 10:59:58.103338] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=490] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:58.103435] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667719, retry interval=1000000, ret=-4179) [2022-08-08 10:59:58.284075] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:58.577026] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:58.577101] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667713, retry interval=1000000, ret=-4179) [2022-08-08 10:59:58.904302] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34695) [2022-08-08 10:59:58.907743] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] [trans sampling] (TRACE=begin_ts=1659927598868959 2022-08-08 02:59:58.868959|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:9895260199439148469, inc:84100504, addr:{ip:"192.168.11.128", port:2882}, t:1659927598866000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC15B-0-0, start_time:1659927598868752, snapshot:1659927598790002|[end_task] u=141 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927598869120|[calc_checksum_by_commit] u=19 ret:0, checksum:1437150504, checksum_log_ts:9223372036854775807|[submit_log] u=67 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=38468 trans_version:1659927598866742, uref:1073741825|[exiting] u=19 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=1 ret:0, log_type:64, log_id:8316396, logging:0, pending:0, uref:1|[end_trans_cb] u=4 ret:0, param:0, param:true|[destroy] u=2 arg1:140085233547952|total_timeu=38746) [2022-08-08 10:59:58.907821] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=39550, time_dist: do_cluster_heartbeat=10, generate_cluster_version=39539) [2022-08-08 10:59:59.020072] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 10:59:59.029609] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927599029600, cur_ts_=1659927599009108, delta=20492) [2022-08-08 10:59:59.065995] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34450) [2022-08-08 10:59:59.069768] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 10:59:59.069790] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 10:59:59.080930] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=50624, time_dist: do_cluster_heartbeat=9, generate_cluster_version=50611) [2022-08-08 10:59:59.086806] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20733) [2022-08-08 10:59:59.089245] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28784, time_dist: generate_cluster_version=28779) [2022-08-08 10:59:59.150701] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927599150689, cur_ts_=1659927599122426, delta=28263) [2022-08-08 10:59:59.222034] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:59.222071] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667718, retry interval=1000000, ret=-4179) [2022-08-08 10:59:59.381164] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927599381158, cur_ts_=1659927599357013, delta=24145) [2022-08-08 10:59:59.486386] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] clock out of order(cur_ts=1659927599486378, cur_ts_=1659927599460661, delta=25717) [2022-08-08 10:59:59.610636] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927599610627, cur_ts_=1659927599567592, delta=43035) [2022-08-08 10:59:59.684943] WARN [LIB] ~ObTimeGuard (utility.h:788) [10868][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard 'ObRpcHandler::process' cost too much time, used=15501, time_dist: 0,1,0,15495,0) [2022-08-08 10:59:59.694084] WARN [RPC.FRAME] process (ob_req_handler.cpp:209) [10868][0][Y0-0000000000000000-0-0] [lt=9133] [dc=0] process handler cost too much time(diff=24642) [2022-08-08 10:59:59.694100] WARN __tg_cleanup (easy_connection.c:3247) [10868][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] easy cost too much time: 24625us server_process [2022-08-08 10:59:59.694107] WARN __tg_cleanup (easy_connection.c:1542) [10868][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] easy cost too much time: 24641us on_readable: 0x7f686b3e0b70 [2022-08-08 10:59:59.694111] WARN ev_invoke_pending (ev.c:2137) [10868][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] epoll handler too much time :24683 count=1 malloc=0/6 write=0/0 read=3/1 server_process=24625/1 client_cb=0/0 [2022-08-08 10:59:59.711216] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 10:59:59.711862] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=496] [dc=0] task process failed, start retry(max retry time=9223372036854667712, retry interval=1000000, ret=-4179) [2022-08-08 10:59:59.757873] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13093) [2022-08-08 10:59:59.758099] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14138, time_dist: do_cluster_heartbeat=9, generate_cluster_version=14127) [2022-08-08 10:59:59.776115] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 10:59:59.776136] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:00.056288] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] clock out of order(cur_ts=1659927600056277, cur_ts_=1659927600025446, delta=30831) [2022-08-08 11:00:00.073112] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:00.077312] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:00.077336] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:00.261754] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:00.261792] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:00.263259] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:00.263305] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667717, retry interval=1000000, ret=-4179) [2022-08-08 11:00:00.388994] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14885600}) [2022-08-08 11:00:00.573923] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927598340996, end_time=1659927600573909, elapsed_time=2232913, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:00:00.748188] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927600748176, cur_ts_=1659927600723384, delta=24792) [2022-08-08 11:00:00.763307] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:00.763356] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667711, retry interval=1000000, ret=-4179) [2022-08-08 11:00:00.773912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:00.784336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:00.974489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:01.079996] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:01.083197] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:01.083249] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:01.087922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:01.147787] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.194031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.254171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.256866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:01.259926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:01.263425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:01.280460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:01.285366] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:01.285645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:01.287874] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:01.287940] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667716, retry interval=1000000, ret=-4179) [2022-08-08 11:00:01.335159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.343337] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:01.371797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.403468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:01.459280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.625684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:01.630085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:01.682527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:01.748020] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927601748001, cur_ts_=1659927601722955, delta=25046) [2022-08-08 11:00:01.794712] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=91803) [2022-08-08 11:00:01.794781] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:15144551035077381457, inc:84100604, addr:"192.168.11.128:2882", t:1659927601691882}, log_id=8316411, used_time=101712) [2022-08-08 11:00:01.795000] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=102687, time_dist: generate_cluster_version=102685) [2022-08-08 11:00:01.843874] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=49121) [2022-08-08 11:00:01.844117] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=23] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:3333805789598801245, inc:84100605, addr:"192.168.11.128:2882", t:1659927601691882}, log_id=8301815, used_time=150362) [2022-08-08 11:00:01.855808] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=162686, time_dist: do_cluster_heartbeat=7, generate_cluster_version=162674) [2022-08-08 11:00:01.884507] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:01.884663] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667710, retry interval=1000000, ret=-4179) [2022-08-08 11:00:01.888028] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34893, time_dist: do_cluster_heartbeat=95, generate_cluster_version=34793) [2022-08-08 11:00:01.889497] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] [trans sampling] (TRACE=begin_ts=1659927601885432 2022-08-08 03:00:01.885432|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:11726979886585825637, inc:84100615, addr:{ip:"192.168.11.128", port:2882}, t:1659927601884000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7DF8-0-0, start_time:1659927601885248, snapshot:1659927601864776|[end_task] u=619 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=25 stc:1659927601886082|[calc_checksum_by_commit] u=42 ret:0, checksum:3247093755, checksum_log_ts:9223372036854775807|[submit_log] u=70 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3221 trans_version:1659927601884408, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:988097, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=4021) [2022-08-08 11:00:02.085311] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:02.088496] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:02.088554] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:02.312571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:02.312818] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:02.314687] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:02.314739] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667715, retry interval=1000000, ret=-4179) [2022-08-08 11:00:02.314952] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10466, time_dist: generate_cluster_version=10463) [2022-08-08 11:00:02.887445] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:02.887507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E3-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:02.921744] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:02.921791] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667709, retry interval=1000000, ret=-4179) [2022-08-08 11:00:03.093296] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:03.096724] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:03.096760] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:03.334845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:03.335118] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:03.336530] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:03.336579] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667714, retry interval=1000000, ret=-4179) [2022-08-08 11:00:03.754374] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:00:03.942259] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:03.942371] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667708, retry interval=1000000, ret=-4179) [2022-08-08 11:00:03.986075] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] clock out of order(cur_ts=1659927603986066, cur_ts_=1659927603957230, delta=28836) [2022-08-08 11:00:04.099167] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:04.107924] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:04.107954] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:04.360552] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:04.360620] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667713, retry interval=1000000, ret=-4179) [2022-08-08 11:00:04.457604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:04.565949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:04.585337] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:04.595531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:04.763397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=239] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:04.774505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:04.775581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=757] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:04.779070] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15985) [2022-08-08 11:00:04.779326] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23619, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23605) [2022-08-08 11:00:04.838378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:04.850006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:04.861900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:04.895187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:04.902527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:04.919956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=272] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:04.930768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:04.946356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:04.951202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:04.951728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:04.951896] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:04.951958] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:04.963170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:04.971963] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:04.972017] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667707, retry interval=1000000, ret=-4179) [2022-08-08 11:00:04.995416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:04.995463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:04.998589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=4744] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.048962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:05.059263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:05.059881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:05.059915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:05.071482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:05.072684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.072959] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.094433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.100727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:05.104050] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.104251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.110821] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:05.114087] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:05.114134] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:05.114806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:05.124737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:05.124788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.137081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.138188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:05.156142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.187716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:05.188241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:05.211578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.219469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=215] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.242467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:05.252684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:05.286026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:05.290405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:05.307333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.379162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:05.385590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.389809] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:05.389915] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667712, retry interval=1000000, ret=-4179) [2022-08-08 11:00:05.399892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.417483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.448532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:05.495449] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=481] [dc=0] clock out of order(cur_ts=1659927605495442, cur_ts_=1659927605464163, delta=31279) [2022-08-08 11:00:05.513592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.567207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:05.650738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.872764] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:05.907044] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14143) [2022-08-08 11:00:05.907347] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15887, time_dist: generate_cluster_version=15884) [2022-08-08 11:00:05.999618] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:05.999683] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667706, retry interval=1000000, ret=-4179) [2022-08-08 11:00:06.030464] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] clock out of order(cur_ts=1659927606030458, cur_ts_=1659927606005499, delta=24959) [2022-08-08 11:00:06.116504] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:06.120679] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:06.120730] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:06.421174] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927606421167, cur_ts_=1659927606395969, delta=25198) [2022-08-08 11:00:06.439075] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:06.439218] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667711, retry interval=1000000, ret=-4179) [2022-08-08 11:00:07.002803] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.002883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E9-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.003559] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E9-0-0] [lt=93] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.003605] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E9-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.003628] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC61E9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.005387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.005431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:07.030818] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:07.031031] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] task process failed, start retry(max retry time=9223372036854667705, retry interval=1000000, ret=-4179) [2022-08-08 11:00:07.122983] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:07.133151] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:07.133180] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:07.140819] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12109) [2022-08-08 11:00:07.143260] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20583, time_dist: do_cluster_heartbeat=9, generate_cluster_version=20564) [2022-08-08 11:00:07.187560] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927607187550, cur_ts_=1659927607156164, delta=31386) [2022-08-08 11:00:07.245922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:07.462241] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:07.462289] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667710, retry interval=1000000, ret=-4179) [2022-08-08 11:00:07.880575] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE311F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:07.880825] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:07.881003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:07.881174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:07.881298] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:07.897224] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=32] [dc=0] [trans sampling] (TRACE=begin_ts=1659927607896413 2022-08-08 03:00:07.896413|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:9103027324332164649, inc:84100842, addr:{ip:"192.168.11.128", port:2882}, t:1659927607895000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A63B9-0-0, start_time:1659927607896215, snapshot:1659927607849220|[end_task] u=136 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927607896567|[calc_checksum_by_commit] u=20 ret:0, checksum:407862623, checksum_log_ts:9223372036854775807|[submit_log] u=63 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=509 trans_version:1659927607896589, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301847, logging:0, pending:0, uref:1|[end_trans_cb] u=3 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=770) [2022-08-08 11:00:08.049658] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] clock out of order(cur_ts=1659927608049647, cur_ts_=1659927608018598, delta=31049) [2022-08-08 11:00:08.109046] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:08.109220] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667704, retry interval=1000000, ret=-4179) [2022-08-08 11:00:08.140939] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:08.156673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:08.156696] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:08.167879] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11724, time_dist: do_cluster_heartbeat=6, generate_cluster_version=11713) [2022-08-08 11:00:08.346414] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:08.353821] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12286) [2022-08-08 11:00:08.354050] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14694, time_dist: generate_cluster_version=14691) [2022-08-08 11:00:08.500903] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:08.501081] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667709, retry interval=1000000, ret=-4179) [2022-08-08 11:00:08.950710] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927608950704, cur_ts_=1659927608927186, delta=23518) [2022-08-08 11:00:08.963535] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12795, time_dist: generate_cluster_version=12792) [2022-08-08 11:00:08.965175] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15287, time_dist: do_cluster_heartbeat=9, generate_cluster_version=15276) [2022-08-08 11:00:09.141092] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:09.141157] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667703, retry interval=1000000, ret=-4179) [2022-08-08 11:00:09.160665] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:09.163705] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:09.163728] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:09.537416] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:09.537453] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667708, retry interval=1000000, ret=-4179) [2022-08-08 11:00:09.786547] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:00:09.786642] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:10.027728] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927610027722, cur_ts_=1659927610002047, delta=25675) [2022-08-08 11:00:10.092726] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14889792}) [2022-08-08 11:00:10.168134] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:10.169902] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:10.170640] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667702, retry interval=1000000, ret=-4179) [2022-08-08 11:00:10.173329] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:10.173367] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:10.293623] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=186] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927608342537, end_time=1659927610293610, elapsed_time=1951073, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:00:10.562522] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:10.562571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667707, retry interval=1000000, ret=-4179) [2022-08-08 11:00:10.830355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=356] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:10.852839] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927610852831, cur_ts_=1659927610756086, delta=96745) [2022-08-08 11:00:10.857893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:10.880662] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=52035) [2022-08-08 11:00:10.880854] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=7] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:110041722609097504, inc:84100951, addr:"192.168.11.128:2882", t:1659927610753519}, log_id=8301861, used_time=124883) [2022-08-08 11:00:10.883676] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=128753, time_dist: generate_cluster_version=128748) [2022-08-08 11:00:10.884853] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:7761595008598954667, inc:84100952, addr:"192.168.11.128:2882", t:1659927610756086}, log_id=8316456, used_time=127332) [2022-08-08 11:00:10.885963] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=129993, time_dist: generate_cluster_version=129988) [2022-08-08 11:00:10.968482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:10.979096] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] clock out of order(cur_ts=1659927610979088, cur_ts_=1659927610907618, delta=71470) [2022-08-08 11:00:10.988804] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19861) [2022-08-08 11:00:10.990284] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=339] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=40964, time_dist: do_cluster_heartbeat=9, generate_cluster_version=40950) [2022-08-08 11:00:10.996322] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45988, time_dist: do_cluster_heartbeat=7, generate_cluster_version=45977) [2022-08-08 11:00:10.999622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:11.108255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:11.164071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.175939] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:11.182452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:11.182485] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:11.201109] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:11.201170] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667701, retry interval=1000000, ret=-4179) [2022-08-08 11:00:11.212322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.275336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:11.276738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=903] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:11.277877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.286097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:11.301390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:11.371446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.375611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:11.396734] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927611396723, cur_ts_=1659927611353660, delta=43063) [2022-08-08 11:00:11.399554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.490158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:11.542529] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=146109) [2022-08-08 11:00:11.542565] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=17, io_time=146135, file_id=237, flush_start_offset=52237254, file_offset=52237871) [2022-08-08 11:00:11.542605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.549464] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:12348773967695975509, inc:84100980, addr:"192.168.11.128:2882", t:1659927611353660}, log_id=8316459, used_time=186884) [2022-08-08 11:00:11.582707] WARN [CLOG] submit_log_succ_cb (ob_log_task.cpp:323) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=33182] [dc=0] transaction on success costs(on_success_duration=33258, pkey={tid:1099511628002, partition_id:0, part_cnt:0}, log_id=8316459) [2022-08-08 11:00:11.587690] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927611587549, cur_ts_=1659927611552041, delta=35508) [2022-08-08 11:00:11.593376] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=1207] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=231328, time_dist: generate_cluster_version=231317) [2022-08-08 11:00:11.598426] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=55840) [2022-08-08 11:00:11.598931] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=75] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:1768304244011423670, inc:84100981, addr:"192.168.11.128:2882", t:1659927611353660}, log_id=8301865, used_time=197909) [2022-08-08 11:00:11.607625] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=208088, time_dist: do_cluster_heartbeat=10, generate_cluster_version=208074) [2022-08-08 11:00:11.640459] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:11.640600] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] task process failed, start retry(max retry time=9223372036854667706, retry interval=1000000, ret=-4179) [2022-08-08 11:00:11.644747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:11.655121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:11.698256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:12.001867] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE312B-0-0] [lt=3834] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:12.001982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE312B-0-0] [lt=50] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:12.002277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:12.002475] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=174] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:12.003027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:12.184291] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:12.187746] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:12.187780] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:12.224221] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:12.224260] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667700, retry interval=1000000, ret=-4179) [2022-08-08 11:00:12.685802] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:12.729237] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927612729232, cur_ts_=1659927612709121, delta=20111) [2022-08-08 11:00:12.735191] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:12.735359] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] task process failed, start retry(max retry time=9223372036854667705, retry interval=1000000, ret=-4179) [2022-08-08 11:00:12.834555] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927612834545, cur_ts_=1659927612781396, delta=53149) [2022-08-08 11:00:13.017252] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE312E-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:13.017279] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE312E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:13.055898] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10008) [2022-08-08 11:00:13.190076] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:13.193688] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:13.193748] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:13.261812] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:13.261874] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667699, retry interval=1000000, ret=-4179) [2022-08-08 11:00:13.407261] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14840, time_dist: generate_cluster_version=14837) [2022-08-08 11:00:13.409834] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13537, time_dist: generate_cluster_version=13534) [2022-08-08 11:00:13.432812] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] clock out of order(cur_ts=1659927613432805, cur_ts_=1659927613409481, delta=23324) [2022-08-08 11:00:13.577201] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14437, time_dist: generate_cluster_version=14432) [2022-08-08 11:00:13.585908] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23884] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11292, time_dist: generate_cluster_version=11287) [2022-08-08 11:00:13.752859] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2900] [dc=0] clock out of order(cur_ts=1659927613752851, cur_ts_=1659927613728428, delta=24423) [2022-08-08 11:00:13.776247] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21996) [2022-08-08 11:00:13.778875] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25883, time_dist: generate_cluster_version=25880) [2022-08-08 11:00:13.801818] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:13.801864] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667704, retry interval=1000000, ret=-4179) [2022-08-08 11:00:13.939167] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=13] [dc=0] [trans sampling] (TRACE=begin_ts=1659927613934804 2022-08-08 03:00:13.934804|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:9950113143564070586, inc:84101071, addr:{ip:"192.168.11.128", port:2882}, t:1659927613924000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=14 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A63D9-0-0, start_time:1659927613934628, snapshot:1659927613853454|[end_task] u=131 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927613934960|[calc_checksum_by_commit] u=21 ret:0, checksum:1894498035, checksum_log_ts:9223372036854775807|[submit_log] u=27 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=4053 trans_version:1659927613934982, uref:1073741825|[exiting] u=24 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301879, logging:0, pending:0, uref:1|[end_trans_cb] u=25 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=4317) [2022-08-08 11:00:14.196983] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:14.200716] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:14.200761] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:14.266812] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:14.266861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:14.283471] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:14.283535] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667698, retry interval=1000000, ret=-4179) [2022-08-08 11:00:14.480304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:14.582672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:14.602989] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:14.614534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:14.782057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:14.793197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:14.800749] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:14.833636] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:14.833706] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667703, retry interval=1000000, ret=-4179) [2022-08-08 11:00:14.855312] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:14.867003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:14.886621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=221] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:14.918472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:14.922362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:14.935617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:14.949734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:14.964495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:14.968039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:14.976826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:14.985234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:15.019104] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:15.019170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.019234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.067493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:15.068200] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3134-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:15.076088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:15.083649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:15.083746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:15.093924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.096872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:15.104311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.115258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.116242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:15.124617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.128094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.135613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:15.146170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.149337] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:15.162547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.162576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:15.175335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.202995] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:15.203367] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:15.206940] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:15.206966] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:15.209050] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:15.227362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.234725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.265728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:15.274394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:15.288128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:15.288191] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:15.305922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:15.310100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:15.312564] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:15.312717] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667697, retry interval=1000000, ret=-4179) [2022-08-08 11:00:15.327669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.399398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:15.406930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.423690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.435382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.466713] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:15.529879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.591845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:15.667440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.838863] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927615838858, cur_ts_=1659927615809124, delta=29734) [2022-08-08 11:00:15.865864] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:15.865916] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667702, retry interval=1000000, ret=-4179) [2022-08-08 11:00:15.891558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:15.962767] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:00:16.209411] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:16.212477] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:16.212519] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:16.343558] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:16.343798] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=183] [dc=0] task process failed, start retry(max retry time=9223372036854667696, retry interval=1000000, ret=-4179) [2022-08-08 11:00:16.892394] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:16.892879] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=358] [dc=0] task process failed, start retry(max retry time=9223372036854667701, retry interval=1000000, ret=-4179) [2022-08-08 11:00:16.968894] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927616967526 2022-08-08 03:00:16.967526|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:16176948570838816779, inc:84101178, addr:{ip:"192.168.11.128", port:2882}, t:1659927616967000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A63E8-0-0, start_time:1659927616967390, snapshot:1659927616877897|[end_task] u=137 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927616967682|[calc_checksum_by_commit] u=22 ret:0, checksum:2687435934, checksum_log_ts:9223372036854775807|[submit_log] u=29 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1049 trans_version:1659927616967704, uref:1073741825|[exiting] u=39 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8301894, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1324) [2022-08-08 11:00:17.024031] WARN [CLOG] submit_log_succ_cb (ob_log_task.cpp:323) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=9] [dc=0] transaction on success costs(on_success_duration=29499, pkey={tid:1099511628002, partition_id:0, part_cnt:0}, log_id=8316488) [2022-08-08 11:00:17.024175] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927617024169, cur_ts_=1659927616993722, delta=30447) [2022-08-08 11:00:17.026151] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34428, time_dist: generate_cluster_version=34423) [2022-08-08 11:00:17.092592] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927617090647 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927617091791 abs_expired_time=1659927619021848 stmt_timeout=1930057 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21594301 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927617085298 start_version=-1 trans_version=1659927617090637 commit_version=0 stmt_start_time=1659927617090637 abs_expired_time=1659927646991038 stmt_timeout=29900401 abs_lock_wait_timeout=1659927646991038 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:10959929626684400484, inc:84101183, addr:"192.168.11.128:2882", t:1659927617090637} checksum_log_ts=9223372036854775807, version=1659927617090647) [2022-08-08 11:00:17.219579] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:17.263874] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=68718) [2022-08-08 11:00:17.296189] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=13] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:14777994576053372144, inc:84101189, addr:"192.168.11.128:2882", t:1659927617187272}, log_id=8316489, used_time=101623) [2022-08-08 11:00:17.298595] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:17.345496] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=153341, time_dist: do_cluster_heartbeat=12, generate_cluster_version=153324) [2022-08-08 11:00:17.357328] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927617357321, cur_ts_=1659927617289545, delta=67776) [2022-08-08 11:00:17.361016] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:17.361040] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:17.408167] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.422353] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=13949] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.422381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.422389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.422474] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.422529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.424349] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C49F2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.429864] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.429940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.433333] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18937) [2022-08-08 11:00:17.434048] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34740, time_dist: do_cluster_heartbeat=9, generate_cluster_version=34727) [2022-08-08 11:00:17.453304] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19915) [2022-08-08 11:00:17.454868] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=208] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45387, time_dist: do_cluster_heartbeat=11, generate_cluster_version=45372) [2022-08-08 11:00:17.465307] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:17.465348] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667695, retry interval=1000000, ret=-4179) [2022-08-08 11:00:17.900567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.900594] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:17.922528] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:17.922601] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667700, retry interval=1000000, ret=-4179) [2022-08-08 11:00:18.398383] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:18.432784] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:18.433859] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=514] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=55152) [2022-08-08 11:00:18.434810] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1591] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=57440, time_dist: generate_cluster_version=57437) [2022-08-08 11:00:18.436782] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=42078, time_dist: generate_cluster_version=42074) [2022-08-08 11:00:18.437923] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:18.437943] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:18.488928] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:18.489236] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667694, retry interval=1000000, ret=-4179) [2022-08-08 11:00:18.635306] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] clock out of order(cur_ts=1659927618635295, cur_ts_=1659927618610666, delta=24629) [2022-08-08 11:00:18.774282] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14863) [2022-08-08 11:00:18.774507] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15889, time_dist: do_cluster_heartbeat=11, generate_cluster_version=15875) [2022-08-08 11:00:18.980480] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:18.980517] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667699, retry interval=1000000, ret=-4179) [2022-08-08 11:00:19.023839] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10634, time_dist: generate_cluster_version=10632) [2022-08-08 11:00:19.173156] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19602) [2022-08-08 11:00:19.173388] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21462, time_dist: generate_cluster_version=21459) [2022-08-08 11:00:19.202943] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17056) [2022-08-08 11:00:19.203184] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18023, time_dist: generate_cluster_version=18020) [2022-08-08 11:00:19.332873] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19141) [2022-08-08 11:00:19.333107] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20214, time_dist: do_cluster_heartbeat=10, generate_cluster_version=20201) [2022-08-08 11:00:19.405844] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18280) [2022-08-08 11:00:19.406094] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19247, time_dist: generate_cluster_version=19245) [2022-08-08 11:00:19.440452] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:19.443389] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:19.443413] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:19.532314] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:19.532364] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667693, retry interval=1000000, ret=-4179) [2022-08-08 11:00:19.560218] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927619560206, cur_ts_=1659927619539428, delta=20778) [2022-08-08 11:00:19.561220] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32465) [2022-08-08 11:00:19.561557] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34991, time_dist: do_cluster_heartbeat=10, generate_cluster_version=34979) [2022-08-08 11:00:19.573843] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12493) [2022-08-08 11:00:19.574056] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13730, time_dist: generate_cluster_version=13729) [2022-08-08 11:00:19.779254] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:00:19.779271] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:19.987221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:19.987387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:20.010905] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=164] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:20.010971] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667698, retry interval=1000000, ret=-4179) [2022-08-08 11:00:20.044076] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927620044009, cur_ts_=1659927620023786, delta=20223) [2022-08-08 11:00:20.138958] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3143-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:20.138977] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3143-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:20.138982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3143-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:20.138987] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3143-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:20.139439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3143-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:20.196912] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14894240}) [2022-08-08 11:00:20.376884] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927618343519, end_time=1659927620376873, elapsed_time=2033354, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:00:20.444867] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:20.449435] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:20.449474] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:20.561120] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927620561115, cur_ts_=1659927620533201, delta=27914) [2022-08-08 11:00:20.579485] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:20.579535] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667692, retry interval=1000000, ret=-4179) [2022-08-08 11:00:20.850388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:20.875423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.018743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.033451] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:21.033502] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667697, retry interval=1000000, ret=-4179) [2022-08-08 11:00:21.129845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:21.184867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.228048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.300507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.300544] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.300559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.311340] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.323356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:21.394231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.394324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=20607] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:21.417329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=6018] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.451744] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:21.454995] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:21.455029] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:21.510789] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.563919] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.583933] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:21.584287] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:21.608489] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:21.608569] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] task process failed, start retry(max retry time=9223372036854667691, retry interval=1000000, ret=-4179) [2022-08-08 11:00:21.670200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:21.670998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:21.722821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:22.058106] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:22.058144] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667696, retry interval=1000000, ret=-4179) [2022-08-08 11:00:22.295164] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927622295154, cur_ts_=1659927622210931, delta=84223) [2022-08-08 11:00:22.412368] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927622412362, cur_ts_=1659927622302026, delta=110336) [2022-08-08 11:00:22.440492] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30266) [2022-08-08 11:00:22.453077] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=45724, time_dist: do_cluster_heartbeat=11, generate_cluster_version=45705) [2022-08-08 11:00:22.457445] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16890) [2022-08-08 11:00:22.458043] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30929, time_dist: do_cluster_heartbeat=9, generate_cluster_version=30915) [2022-08-08 11:00:22.459300] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:22.464526] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:22.464680] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:22.538185] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927622538177, cur_ts_=1659927622510681, delta=27496) [2022-08-08 11:00:22.611567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.611602] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.611859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.611895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.611975] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.612397] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.612420] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.612554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6201-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:22.651926] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:22.652008] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667690, retry interval=1000000, ret=-4179) [2022-08-08 11:00:22.988534] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927622988525, cur_ts_=1659927622920039, delta=68486) [2022-08-08 11:00:23.104846] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:23.104926] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667695, retry interval=1000000, ret=-4179) [2022-08-08 11:00:23.468667] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:23.472148] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:23.472183] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:23.675173] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:23.675212] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667689, retry interval=1000000, ret=-4179) [2022-08-08 11:00:24.098721] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12444, time_dist: generate_cluster_version=12439) [2022-08-08 11:00:24.127073] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:24.127136] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667694, retry interval=1000000, ret=-4179) [2022-08-08 11:00:24.434419] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] clock out of order(cur_ts=1659927624434409, cur_ts_=1659927624413406, delta=21003) [2022-08-08 11:00:24.474906] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:24.482525] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=141] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:24.482565] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:24.503371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:24.604029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:24.619368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:24.629764] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=1169] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:24.680483] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A3C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:24.680609] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A3C-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:24.681213] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A3C-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:24.711920] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:24.711971] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667688, retry interval=1000000, ret=-4179) [2022-08-08 11:00:24.806987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:24.817273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:24.817469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:24.880231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:24.889847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:24.916802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:24.919671] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] clock out of order(cur_ts=1659927624919662, cur_ts_=1659927624889195, delta=30467) [2022-08-08 11:00:24.943059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:24.943121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:24.953311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:24.974214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=543] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:24.983975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:24.986388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.019964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:25.025471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.026327] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=3003] [dc=0] clock out of order(cur_ts=1659927625026312, cur_ts_=1659927624991318, delta=34994) [2022-08-08 11:00:25.034614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.046161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.046200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:25.092490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=228] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:25.098001] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:25.099076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:25.107009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:25.107723] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11627, time_dist: do_cluster_heartbeat=7, generate_cluster_version=11617) [2022-08-08 11:00:25.118103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.120027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:25.124049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.140226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:25.141781] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.146388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.152103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.155282] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:25.155348] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667693, retry interval=1000000, ret=-4179) [2022-08-08 11:00:25.159540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:25.171219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.171267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:25.178025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=163] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.182857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=450] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:25.191722] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.224503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:25.228643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:25.250270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.256249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.287501] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:25.295273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:25.323908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:25.334870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:25.347074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.420896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:25.431323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.449225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.454682] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.487909] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:25.488611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:25.491423] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:25.491460] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:25.497806] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14427, time_dist: generate_cluster_version=14423) [2022-08-08 11:00:25.546688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.612330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:25.700932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:25.716397] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:25.716500] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:25.737714] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:25.737754] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667687, retry interval=1000000, ret=-4179) [2022-08-08 11:00:25.910275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:26.029633] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927626029628, cur_ts_=1659927626006208, delta=23420) [2022-08-08 11:00:26.170992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:26.171134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:26.198986] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:26.199210] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667692, retry interval=1000000, ret=-4179) [2022-08-08 11:00:26.497189] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:26.500227] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:26.500263] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:26.762647] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:26.762724] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667686, retry interval=1000000, ret=-4179) [2022-08-08 11:00:26.860443] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927626860433, cur_ts_=1659927626839689, delta=20744) [2022-08-08 11:00:26.951422] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=273] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=37461, time_dist: generate_cluster_version=37457) [2022-08-08 11:00:26.957153] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15230, time_dist: generate_cluster_version=15227) [2022-08-08 11:00:27.203653] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.203697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.224163] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:27.224234] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667691, retry interval=1000000, ret=-4179) [2022-08-08 11:00:27.323439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:27.496901] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] clock out of order(cur_ts=1659927627496892, cur_ts_=1659927627472587, delta=24305) [2022-08-08 11:00:27.515040] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:27.526503] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29186) [2022-08-08 11:00:27.526665] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=16] [dc=0] flush_cb cost too much time(partition_key_={tid:1100611139404002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8301946, proposal_id:{time_to_usec:1659815624166414, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:52386124, size:573}, after_consume_timestamp:1659927627472587}, end_time=1659927627517976, begin_time=1659927627472587, cost_time=45389) [2022-08-08 11:00:27.549544] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=54130, time_dist: do_cluster_heartbeat=12, generate_cluster_version=54114) [2022-08-08 11:00:27.550642] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:27.557521] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=6856] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:27.559461] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32896) [2022-08-08 11:00:27.559838] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=61842, time_dist: do_cluster_heartbeat=11, generate_cluster_version=61829) [2022-08-08 11:00:27.765307] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.765643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=122] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.765791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.765968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.766048] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=69] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.766203] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.766495] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.766569] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4B80-0-0] [lt=46] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:27.785083] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:27.785117] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667685, retry interval=1000000, ret=-4179) [2022-08-08 11:00:28.083427] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] clock out of order(cur_ts=1659927628083418, cur_ts_=1659927628039466, delta=43952) [2022-08-08 11:00:28.228787] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE315B-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:28.256858] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:28.256912] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667690, retry interval=1000000, ret=-4179) [2022-08-08 11:00:28.354962] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10983) [2022-08-08 11:00:28.355826] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12908, time_dist: generate_cluster_version=12905) [2022-08-08 11:00:28.520157] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19803) [2022-08-08 11:00:28.520962] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21473, time_dist: do_cluster_heartbeat=9, generate_cluster_version=21461) [2022-08-08 11:00:28.527043] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:28.528998] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18539, time_dist: do_cluster_heartbeat=16, generate_cluster_version=18499) [2022-08-08 11:00:28.570467] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:28.633461] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927628633453, cur_ts_=1659927628576158, delta=57295) [2022-08-08 11:00:28.691356] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=157493) [2022-08-08 11:00:28.691387] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=11, io_time=157515, file_id=237, flush_start_offset=52396438, file_offset=52396964) [2022-08-08 11:00:28.691448] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=23] [dc=0] transaction log sync use too much time(partition={tid:1099511628085, partition_id:0, part_cnt:0}, trans_id={hash:4059523010321673636, inc:84101619, addr:"192.168.11.128:2882", t:1659927628532713}, log_id=1160694, used_time=157745) [2022-08-08 11:00:28.698725] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:28.698756] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:28.749456] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] clock out of order(cur_ts=1659927628749445, cur_ts_=1659927628693283, delta=56162) [2022-08-08 11:00:28.756040] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=63557) [2022-08-08 11:00:28.813727] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:28.813764] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667684, retry interval=1000000, ret=-4179) [2022-08-08 11:00:29.151918] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927629151905, cur_ts_=1659927629126281, delta=25624) [2022-08-08 11:00:29.231648] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16047) [2022-08-08 11:00:29.281091] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927629281085, cur_ts_=1659927629254486, delta=26599) [2022-08-08 11:00:29.346617] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:29.346654] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667689, retry interval=1000000, ret=-4179) [2022-08-08 11:00:29.759340] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:29.762697] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927629762690, cur_ts_=1659927629740636, delta=22054) [2022-08-08 11:00:29.766097] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:29.766136] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:29.778334] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:00:29.778352] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:29.839253] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:29.839289] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667683, retry interval=1000000, ret=-4179) [2022-08-08 11:00:29.919966] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] clock out of order(cur_ts=1659927629919956, cur_ts_=1659927629893304, delta=26652) [2022-08-08 11:00:30.061810] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=112237) [2022-08-08 11:00:30.061958] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:13565848541612775018, inc:84101673, addr:"192.168.11.128:2882", t:1659927629933001}, log_id=8301958, used_time=112465) [2022-08-08 11:00:30.062133] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=113436, time_dist: generate_cluster_version=113434) [2022-08-08 11:00:30.076181] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] slow flush(flush_time=14311, io_time=112297, file_id=237, flush_start_offset=52410926, file_offset=52411548) [2022-08-08 11:00:30.081421] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:5046016434982779395, inc:84101674, addr:"192.168.11.128:2882", t:1659927629933001}, log_id=8316553, used_time=131099) [2022-08-08 11:00:30.081637] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=131835, time_dist: generate_cluster_version=131833) [2022-08-08 11:00:30.356220] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14898592}) [2022-08-08 11:00:30.356757] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14210) [2022-08-08 11:00:30.357466] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15718, time_dist: generate_cluster_version=15715) [2022-08-08 11:00:30.387009] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24486) [2022-08-08 11:00:30.391655] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32080, time_dist: generate_cluster_version=32078) [2022-08-08 11:00:30.392857] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:30.392908] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667688, retry interval=1000000, ret=-4179) [2022-08-08 11:00:30.551235] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927628343859, end_time=1659927630551222, elapsed_time=2207363, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:00:30.605163] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] clock out of order(cur_ts=1659927630605156, cur_ts_=1659927630571561, delta=33595) [2022-08-08 11:00:30.767779] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:30.770889] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:30.770924] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:30.866497] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:30.866537] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667682, retry interval=1000000, ret=-4179) [2022-08-08 11:00:30.866599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:30.896282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.052881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.150570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:31.202395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.247816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.316938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.317024] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.319365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.327481] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.348189] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:31.395612] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A45-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.395709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A45-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.396010] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A45-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.396179] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A45-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.396551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A45-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.396776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A45-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.411811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.414097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:31.424297] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:31.424415] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667687, retry interval=1000000, ret=-4179) [2022-08-08 11:00:31.436540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.557818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.558839] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927631558832, cur_ts_=1659927631513000, delta=45832) [2022-08-08 11:00:31.579896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.630178] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=148] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=109330) [2022-08-08 11:00:31.630211] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] slow flush(flush_time=13, io_time=109354, file_id=237, flush_start_offset=52427012, file_offset=52427629) [2022-08-08 11:00:31.630461] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=17] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:18125918100872515098, inc:84101733, addr:"192.168.11.128:2882", t:1659927631513001}, log_id=8316562, used_time=109722) [2022-08-08 11:00:31.630603] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=110604, time_dist: do_cluster_heartbeat=11, generate_cluster_version=110592) [2022-08-08 11:00:31.637926] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1773] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=78705, time_dist: do_cluster_heartbeat=8, generate_cluster_version=78693) [2022-08-08 11:00:31.688105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:31.693922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:31.741391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:31.775701] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:31.779117] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:31.779156] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:31.871028] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:31.893438] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:31.893504] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667681, retry interval=1000000, ret=-4179) [2022-08-08 11:00:31.933440] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=4315] [dc=0] clock out of order(cur_ts=1659927631933435, cur_ts_=1659927631911702, delta=21733) [2022-08-08 11:00:32.111156] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927632110305 2022-08-08 03:00:32.110305|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:12768308287855619011, inc:84101752, addr:{ip:"192.168.11.128", port:2882}, t:1659927632108000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6434-0-0, start_time:1659927632110189, snapshot:1659927632012308|[end_task] u=168 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927632110491|[calc_checksum_by_commit] u=20 ret:0, checksum:2568799828, checksum_log_ts:9223372036854775807|[submit_log] u=42 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=529 trans_version:1659927632110512, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301970, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=813) [2022-08-08 11:00:32.267955] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3167-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.268005] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3167-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.268139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3167-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.268427] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3167-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.268673] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.269181] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.269217] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:32.330047] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10441, time_dist: print_stat=9, generate_cluster_version=10423, cluster_service_self_check=4) [2022-08-08 11:00:32.341931] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19683, time_dist: do_cluster_heartbeat=8, generate_cluster_version=19670) [2022-08-08 11:00:32.386461] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1096] [dc=0] clock out of order(cur_ts=1659927632386452, cur_ts_=1659927632341833, delta=44619) [2022-08-08 11:00:32.485307] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:32.485412] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667686, retry interval=1000000, ret=-4179) [2022-08-08 11:00:32.686383] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:32.783124] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:32.786311] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:32.786370] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:32.906800] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:32.906949] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:32.928307] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:32.928357] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667680, retry interval=1000000, ret=-4179) [2022-08-08 11:00:33.515843] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:33.515880] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667685, retry interval=1000000, ret=-4179) [2022-08-08 11:00:33.788641] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:33.793276] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:33.793315] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:33.949377] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:33.949447] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667679, retry interval=1000000, ret=-4179) [2022-08-08 11:00:34.192385] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11130) [2022-08-08 11:00:34.192796] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12346, time_dist: generate_cluster_version=12343) [2022-08-08 11:00:34.279906] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE316D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:34.280097] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE316D-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:34.280828] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE316D-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:34.280861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE316D-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:34.280970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:34.281742] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:34.416599] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927634416592, cur_ts_=1659927634393726, delta=22866) [2022-08-08 11:00:34.528504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:34.543324] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18067, time_dist: generate_cluster_version=18064) [2022-08-08 11:00:34.578563] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=299] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:34.578649] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667684, retry interval=1000000, ret=-4179) [2022-08-08 11:00:34.623956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:34.644233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:34.647633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:34.795730] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:34.798855] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:34.798891] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:34.825355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:34.837505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:34.839700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=195] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:34.901924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:34.909436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=29998] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:34.941106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=5869] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:34.956677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:34.962386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:34.967589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:34.974401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:34.979539] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:34.979574] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667678, retry interval=1000000, ret=-4179) [2022-08-08 11:00:34.998056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.003156] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.008841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:35.039688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:35.042397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.059468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.068110] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:35.068277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.108296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:35.111362] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] [trans sampling] (TRACE=begin_ts=1659927635110479 2022-08-08 03:00:35.110479|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:6193647632285394224, inc:84101869, addr:{ip:"192.168.11.128", port:2882}, t:1659927635109000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6443-0-0, start_time:1659927635110360, snapshot:1659927635039015|[end_task] u=176 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927635110673|[calc_checksum_by_commit] u=21 ret:0, checksum:4241071778, checksum_log_ts:9223372036854775807|[submit_log] u=43 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=551 trans_version:1659927635110695, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8301985, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=845) [2022-08-08 11:00:35.115656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:35.120760] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:35.131056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:35.136194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.141416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:35.147529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.162806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.162931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:35.162966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.167664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.182686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:35.191142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.193350] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:35.198653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.206394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:35.208966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.243422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:35.249022] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:35.266018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.276937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.308171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:35.312062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:35.339533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:35.360256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:35.370663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.437010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:35.448040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.465322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.476556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.503832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:35.570265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.581115] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A4B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:35.606227] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:35.606265] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667683, retry interval=1000000, ret=-4179) [2022-08-08 11:00:35.629023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:35.722358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.801620] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:35.809031] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:35.809071] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:35.924734] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10670) [2022-08-08 11:00:35.924947] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11756, time_dist: do_cluster_heartbeat=7, generate_cluster_version=11748) [2022-08-08 11:00:35.930694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:35.990806] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=282] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:00:36.014210] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:36.014260] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667677, retry interval=1000000, ret=-4179) [2022-08-08 11:00:36.143933] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13482) [2022-08-08 11:00:36.144178] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14685, time_dist: generate_cluster_version=14683) [2022-08-08 11:00:36.631553] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:36.631608] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667682, retry interval=1000000, ret=-4179) [2022-08-08 11:00:36.811455] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:36.814531] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:36.814566] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:37.037103] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:37.037145] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667676, retry interval=1000000, ret=-4179) [2022-08-08 11:00:37.347353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:37.658948] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:37.659006] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667681, retry interval=1000000, ret=-4179) [2022-08-08 11:00:37.819392] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927637819380, cur_ts_=1659927637789838, delta=29542) [2022-08-08 11:00:37.819480] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:37.828481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:37.828608] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:38.067240] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=553] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:38.067327] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667675, retry interval=1000000, ret=-4179) [2022-08-08 11:00:38.145017] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927638145008, cur_ts_=1659927638115619, delta=29389) [2022-08-08 11:00:38.151041] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14462, time_dist: generate_cluster_version=14458) [2022-08-08 11:00:38.324614] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10204, time_dist: generate_cluster_version=10200) [2022-08-08 11:00:38.689329] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:38.689389] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667680, retry interval=1000000, ret=-4179) [2022-08-08 11:00:38.752891] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17333, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17321) [2022-08-08 11:00:38.808437] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:38.826363] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10786) [2022-08-08 11:00:38.831353] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:38.835390] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:38.835419] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:39.094969] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:39.096129] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667674, retry interval=1000000, ret=-4179) [2022-08-08 11:00:39.160362] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29535, time_dist: generate_cluster_version=29531) [2022-08-08 11:00:39.162049] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22360, time_dist: generate_cluster_version=22356) [2022-08-08 11:00:39.329290] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE317C-0-0] [lt=19] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927639323086 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927639323076 abs_expired_time=1659927641258658 stmt_timeout=1935582 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21594617 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927639278357 start_version=-1 trans_version=1659927639323076 commit_version=0 stmt_start_time=1659927639314744 abs_expired_time=1659927669216069 stmt_timeout=29901325 abs_lock_wait_timeout=1659927669216069 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:15592247653451707183, inc:84102033, addr:"192.168.11.128:2882", t:1659927639314744} checksum_log_ts=9223372036854775807, version=1659927639323086) [2022-08-08 11:00:39.544973] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927639544966, cur_ts_=1659927639523936, delta=21030) [2022-08-08 11:00:39.731529] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927639731524, cur_ts_=1659927639703105, delta=28419) [2022-08-08 11:00:39.783949] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:39.784002] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667679, retry interval=1000000, ret=-4179) [2022-08-08 11:00:39.789022] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:00:39.789049] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:39.805497] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20414) [2022-08-08 11:00:39.810010] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18208, time_dist: do_cluster_heartbeat=11, generate_cluster_version=18192) [2022-08-08 11:00:39.842153] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=1716] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:39.887393] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927639887384, cur_ts_=1659927639842753, delta=44631) [2022-08-08 11:00:39.889991] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:39.890021] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:39.913794] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19053, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19040) [2022-08-08 11:00:40.030104] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14903040}) [2022-08-08 11:00:40.134142] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:40.134442] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667673, retry interval=1000000, ret=-4179) [2022-08-08 11:00:40.135694] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20685, time_dist: do_cluster_heartbeat=10, generate_cluster_version=20672) [2022-08-08 11:00:40.247362] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927638344679, end_time=1659927640247349, elapsed_time=1902670, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:00:40.349967] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.350058] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.350229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.351278] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.351318] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.351329] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.351333] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.353753] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.353780] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.353982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.354137] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE317F-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.354253] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.354596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:40.813439] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:40.813502] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667678, retry interval=1000000, ret=-4179) [2022-08-08 11:00:40.888140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:40.896713] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:40.901610] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:40.901658] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:40.920720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.070915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.126804] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=15] [dc=0] [trans sampling] (TRACE=begin_ts=1659927641125783 2022-08-08 03:00:41.125783|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:3744232410359877925, inc:84102100, addr:{ip:"192.168.11.128", port:2882}, t:1659927641124000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6461-0-0, start_time:1659927641125605, snapshot:1659927641088097|[end_task] u=145 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927641125947|[calc_checksum_by_commit] u=22 ret:0, checksum:3508141124, checksum_log_ts:9223372036854775807|[submit_log] u=48 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=701 trans_version:1659927641125970, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8302015, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=977) [2022-08-08 11:00:41.139483] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:41.139561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:41.159361] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:41.159406] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667672, retry interval=1000000, ret=-4179) [2022-08-08 11:00:41.171497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:41.223437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.266553] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.334508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.337818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.339041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.348016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.367325] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE3182-0-0] [lt=5] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927641356192 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927641366523 abs_expired_time=1659927643296700 stmt_timeout=1930177 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21594647 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927641345344 start_version=-1 trans_version=1659927641356182 commit_version=0 stmt_start_time=1659927641347916 abs_expired_time=1659927671248642 stmt_timeout=29900726 abs_lock_wait_timeout=1659927671248642 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:17019898248853707520, inc:84102111, addr:"192.168.11.128:2882", t:1659927641347916} checksum_log_ts=9223372036854775807, version=1659927641356192) [2022-08-08 11:00:41.367386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:41.427879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.436203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:41.452191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.576562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.605700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25990] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.655400] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927641655393, cur_ts_=1659927641623148, delta=32245) [2022-08-08 11:00:41.711485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:41.716607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:41.760776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:41.839303] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:41.839476] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] task process failed, start retry(max retry time=9223372036854667677, retry interval=1000000, ret=-4179) [2022-08-08 11:00:41.886220] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927641886213, cur_ts_=1659927641861716, delta=24497) [2022-08-08 11:00:41.903892] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:41.907242] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:41.907279] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:42.179708] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927642179696, cur_ts_=1659927642030534, delta=149162) [2022-08-08 11:00:42.241838] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=5295] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:42.242578] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667671, retry interval=1000000, ret=-4179) [2022-08-08 11:00:42.662103] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927642662098, cur_ts_=1659927642638484, delta=23614) [2022-08-08 11:00:42.846967] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12452) [2022-08-08 11:00:42.847275] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14922, time_dist: do_cluster_heartbeat=11, generate_cluster_version=14908) [2022-08-08 11:00:42.861924] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:42.862093] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:42.893045] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:42.893104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667676, retry interval=1000000, ret=-4179) [2022-08-08 11:00:42.908519] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14713) [2022-08-08 11:00:42.909498] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:42.913741] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:42.913780] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:42.974933] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927642974925, cur_ts_=1659927642953414, delta=21511) [2022-08-08 11:00:43.268505] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:43.268553] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667670, retry interval=1000000, ret=-4179) [2022-08-08 11:00:43.409320] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927643409313, cur_ts_=1659927643362062, delta=47251) [2022-08-08 11:00:43.490269] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27516) [2022-08-08 11:00:43.492831] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30852, time_dist: print_stat=8, generate_cluster_version=30832, cluster_service_self_check=7) [2022-08-08 11:00:43.515051] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24718) [2022-08-08 11:00:43.518026] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=46908, time_dist: do_cluster_heartbeat=11, generate_cluster_version=46892) [2022-08-08 11:00:43.537302] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17063) [2022-08-08 11:00:43.599519] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927643599509, cur_ts_=1659927643575447, delta=24062) [2022-08-08 11:00:43.903518] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.903537] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.903543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.903547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.904171] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.904203] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.904247] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.904264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.904270] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.904286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A57-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:43.906495] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15076) [2022-08-08 11:00:43.908165] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17941, time_dist: generate_cluster_version=17934) [2022-08-08 11:00:43.942154] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:43.942192] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667675, retry interval=1000000, ret=-4179) [2022-08-08 11:00:43.952380] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:43.956114] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:43.956157] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:44.129724] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927644128785 2022-08-08 03:00:44.128785|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:6321151177517613431, inc:84102211, addr:{ip:"192.168.11.128", port:2882}, t:1659927644127000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=5 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC23F-0-0, start_time:1659927644128674, snapshot:1659927644113150|[end_task] u=226 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927644129028|[calc_checksum_by_commit] u=20 ret:0, checksum:377768129, checksum_log_ts:9223372036854775807|[submit_log] u=42 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=550 trans_version:1659927644127950, uref:1073741825|[exiting] u=19 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8316624, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=901) [2022-08-08 11:00:44.274509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:44.274577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:44.295232] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:44.295268] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667669, retry interval=1000000, ret=-4179) [2022-08-08 11:00:44.519924] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318B-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.519983] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318B-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.520000] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318B-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.520199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.520357] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.520704] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.520723] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:44.552913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:44.648520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=2109] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:44.667153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:44.669675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:44.844267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:44.856383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:44.860064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:44.926173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:44.929801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:44.959129] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:44.963662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:44.966642] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:44.966697] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:44.982391] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:44.982445] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667674, retry interval=1000000, ret=-4179) [2022-08-08 11:00:44.983443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:44.991558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:44.996496] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.014550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.027896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.049904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:45.066070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=153] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:45.068574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.090938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.091739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=130] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:45.091819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.123726] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:45.138988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:45.145002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:45.155983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:45.156387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.166280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:45.166634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.178012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.192813] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:45.192866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.192869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.203204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:45.215558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.215890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:45.216398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.227671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.229313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:45.268300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:45.269569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:45.271352] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=22] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511628002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8316630, proposal_id:{time_to_usec:1659815620290075, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:52550066, size:568}, after_consume_timestamp:1659927645258612}, end_time=1659927645271265, begin_time=1659927645258612, cost_time=12653) [2022-08-08 11:00:45.289356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.301060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.318166] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:45.318229] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667668, retry interval=1000000, ret=-4179) [2022-08-08 11:00:45.324013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:45.334310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:45.362641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=135] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:45.375822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:45.393743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.462296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:45.473208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.486579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.494015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=641] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.526974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:45.529147] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.529178] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318E-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.529373] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.529484] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE318E-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.529612] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.529902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.530090] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.530260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:45.666750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:45.668888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.686023] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927645686015, cur_ts_=1659927645623843, delta=62172) [2022-08-08 11:00:45.744452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.951258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:45.969082] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:45.972395] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:45.972434] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:46.009688] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:46.009870] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667673, retry interval=1000000, ret=-4179) [2022-08-08 11:00:46.206653] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=303] [dc=0] clock out of order(cur_ts=1659927646206644, cur_ts_=1659927646173463, delta=33181) [2022-08-08 11:00:46.373360] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:46.373404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667667, retry interval=1000000, ret=-4179) [2022-08-08 11:00:46.536414] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:46.536547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:46.536567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:46.976153] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:46.980362] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:46.980400] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:47.032492] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:47.032531] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667672, retry interval=1000000, ret=-4179) [2022-08-08 11:00:47.294738] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=42] [dc=0] [trans sampling] (TRACE=begin_ts=1659927647291382 2022-08-08 03:00:47.291382|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:7313265161169640810, inc:84102322, addr:{ip:"192.168.11.128", port:2882}, t:1659927647290000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC24F-0-0, start_time:1659927647291155, snapshot:1659927647287210|[end_task] u=1889 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=16 stc:1659927647293293|[calc_checksum_by_commit] u=65 ret:0, checksum:2737025757, checksum_log_ts:9223372036854775807|[submit_log] u=56 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=28 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1160 trans_version:1659927647293282, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8316640, logging:0, pending:0, uref:1|[end_trans_cb] u=72 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=3318) [2022-08-08 11:00:47.371479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:47.400510] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:47.400553] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667666, retry interval=1000000, ret=-4179) [2022-08-08 11:00:47.982865] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:47.986420] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:47.986468] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:48.053116] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:48.053213] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667671, retry interval=1000000, ret=-4179) [2022-08-08 11:00:48.472235] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927648472226, cur_ts_=1659927648369603, delta=102623) [2022-08-08 11:00:48.509503] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:48.509553] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:48.560486] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:48.560526] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667665, retry interval=1000000, ret=-4179) [2022-08-08 11:00:48.572911] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16321) [2022-08-08 11:00:48.574099] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20177, time_dist: do_cluster_heartbeat=9, generate_cluster_version=20164) [2022-08-08 11:00:48.587395] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11629) [2022-08-08 11:00:48.587612] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18425, time_dist: do_cluster_heartbeat=9, generate_cluster_version=18414) [2022-08-08 11:00:48.760738] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927648760731, cur_ts_=1659927648718091, delta=42640) [2022-08-08 11:00:48.781319] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20614) [2022-08-08 11:00:48.781649] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38277, time_dist: generate_cluster_version=38267) [2022-08-08 11:00:48.787787] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30987, time_dist: do_cluster_heartbeat=9, generate_cluster_version=30974) [2022-08-08 11:00:48.863917] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927648863899, cur_ts_=1659927648799763, delta=64136) [2022-08-08 11:00:48.903102] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:48.989141] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:48.992117] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:48.992156] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:49.059129] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:49.059157] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:49.086809] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:49.086876] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667670, retry interval=1000000, ret=-4179) [2022-08-08 11:00:49.189774] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] clock out of order(cur_ts=1659927649189766, cur_ts_=1659927649166653, delta=23113) [2022-08-08 11:00:49.582529] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:49.582667] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667664, retry interval=1000000, ret=-4179) [2022-08-08 11:00:49.603046] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE319A-0-0] [lt=3] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927649591754 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927649591744 abs_expired_time=1659927651532302 stmt_timeout=1940558 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21594757 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927649575255 start_version=-1 trans_version=1659927649591744 commit_version=0 stmt_start_time=1659927649575245 abs_expired_time=1659927679482413 stmt_timeout=29907168 abs_lock_wait_timeout=1659927679482413 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:5163571105478942789, inc:84102418, addr:"192.168.11.128:2882", t:1659927649575245} checksum_log_ts=9223372036854775807, version=1659927649591754) [2022-08-08 11:00:49.612665] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20484) [2022-08-08 11:00:49.632847] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17720) [2022-08-08 11:00:49.780434] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:00:49.780464] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:49.996031] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:49.999863] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:49.999924] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:50.128212] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:50.128498] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=198] [dc=0] task process failed, start retry(max retry time=9223372036854667669, retry interval=1000000, ret=-4179) [2022-08-08 11:00:50.295342] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14907328}) [2022-08-08 11:00:50.329033] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927650329021, cur_ts_=1659927650294110, delta=34911) [2022-08-08 11:00:50.544802] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927648346201, end_time=1659927650544788, elapsed_time=2198587, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:00:50.587762] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.587782] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.587837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.587969] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.588044] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.588102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.588222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.588345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.588513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.589176] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A24-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.596046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.596110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:50.614225] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:50.614262] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667663, retry interval=1000000, ret=-4179) [2022-08-08 11:00:50.619014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE319D-0-0] [lt=1445] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619074] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE319D-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619094] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE319D-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE319D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619219] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619327] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619465] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.619617] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:50.905360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:50.937550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.001962] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:51.005256] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:51.005307] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:51.087700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.137716] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927651137707, cur_ts_=1659927651106033, delta=31674) [2022-08-08 11:00:51.141741] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BA2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:51.141830] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BA2-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:51.141869] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BA2-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:51.170160] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:51.170198] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667668, retry interval=1000000, ret=-4179) [2022-08-08 11:00:51.191031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:51.247801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.284279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.357194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.359530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.361520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.363521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.388480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:51.450921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.455229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:51.470094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.594851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.624246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.628837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A0-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.628912] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A0-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.629301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A0-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.629446] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.629552] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.629616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.630217] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:51.642300] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:51.642350] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667662, retry interval=1000000, ret=-4179) [2022-08-08 11:00:51.731245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:51.732861] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:51.783492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:52.009733] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:52.013476] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:52.013515] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:52.194431] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:52.195770] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854667667, retry interval=1000000, ret=-4179) [2022-08-08 11:00:52.649054] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:52.649082] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:52.668897] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:52.668959] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667661, retry interval=1000000, ret=-4179) [2022-08-08 11:00:52.686751] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:53.016593] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:53.019825] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:53.019879] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:53.217345] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:53.217435] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667666, retry interval=1000000, ret=-4179) [2022-08-08 11:00:53.422652] WARN [SQL] execute_get_plan (ob_sql.cpp:3081) [11024][0][YB42C0A80B80-0005E3CF9BC83600-0-0] [lt=16] [dc=0] fail to get plan retry(ret=-5138) [2022-08-08 11:00:53.483488] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] clock out of order(cur_ts=1659927653483478, cur_ts_=1659927653455508, delta=27970) [2022-08-08 11:00:53.484338] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC83600-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 11:00:53.489775] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC83600-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 11:00:53.489885] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [11024][0][YB42C0A80B80-0005E3CF9BC83600-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Success") [2022-08-08 11:00:53.691922] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:53.691975] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667660, retry interval=1000000, ret=-4179) [2022-08-08 11:00:54.023044] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:54.027550] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:54.027574] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:54.222971] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:00:54.240107] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:54.240232] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667665, retry interval=1000000, ret=-4179) [2022-08-08 11:00:54.574300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:54.594756] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10135, time_dist: generate_cluster_version=10132) [2022-08-08 11:00:54.652823] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A9-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:54.652871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A9-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:54.653058] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:54.653187] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31A9-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:54.664848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:54.686627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:54.687690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:54.719531] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:54.719621] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] task process failed, start retry(max retry time=9223372036854667659, retry interval=1000000, ret=-4179) [2022-08-08 11:00:54.869272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:54.879726] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:54.879931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:54.951389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:54.951428] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=292] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:54.987928] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927654987917, cur_ts_=1659927654961045, delta=26872) [2022-08-08 11:00:54.992009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.004741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:55.005888] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15815, time_dist: do_cluster_heartbeat=8, generate_cluster_version=15803) [2022-08-08 11:00:55.008931] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16746, time_dist: generate_cluster_version=16741) [2022-08-08 11:00:55.010672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:55.014980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.030754] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:55.037957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.043964] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:55.044009] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:55.044377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.065881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.091143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:55.092327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.107500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.107557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.107825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:55.139559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:00:55.154175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:55.164645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:55.174534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.176868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:55.185646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.185748] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.195326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.208135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.216384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.216585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:55.228286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.238421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:55.238657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=199] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.241618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.248592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.251837] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.269138] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:55.269200] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667664, retry interval=1000000, ret=-4179) [2022-08-08 11:00:55.284944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:55.285035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=470] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.306694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.322109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.342385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:55.353452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:00:55.386091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:00:55.396080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:55.412248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.482891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:55.492157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.502574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.514287] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.548149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:00:55.623400] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11095) [2022-08-08 11:00:55.623651] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11955, time_dist: generate_cluster_version=11952) [2022-08-08 11:00:55.624146] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13083, time_dist: do_cluster_heartbeat=11, generate_cluster_version=13065) [2022-08-08 11:00:55.658125] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31AC-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:55.658352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:55.684243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:00:55.693725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.765287] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:55.797314] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:55.797369] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667658, retry interval=1000000, ret=-4179) [2022-08-08 11:00:55.986604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:00:56.028559] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:00:56.034235] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13337, time_dist: generate_cluster_version=13334) [2022-08-08 11:00:56.064480] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:56.068120] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:56.068161] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:56.298236] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:56.298303] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667663, retry interval=1000000, ret=-4179) [2022-08-08 11:00:56.660782] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24055) [2022-08-08 11:00:56.661019] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25016, time_dist: generate_cluster_version=25014) [2022-08-08 11:00:56.689622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:00:56.819225] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:56.819429] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] task process failed, start retry(max retry time=9223372036854667657, retry interval=1000000, ret=-4179) [2022-08-08 11:00:57.075898] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:57.079465] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:57.079508] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:57.321062] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:57.321112] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667662, retry interval=1000000, ret=-4179) [2022-08-08 11:00:57.396576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:00:57.694927] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE31B2-0-0] [lt=25] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927657689779 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927657694396 abs_expired_time=1659927659623701 stmt_timeout=1929305 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21594868 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927657633214 start_version=-1 trans_version=1659927657689769 commit_version=0 stmt_start_time=1659927657674869 abs_expired_time=1659927687577334 stmt_timeout=29902465 abs_lock_wait_timeout=1659927687577334 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:3238136037801491479, inc:84102722, addr:"192.168.11.128:2882", t:1659927657674869} checksum_log_ts=9223372036854775807, version=1659927657689779) [2022-08-08 11:00:57.703771] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12889) [2022-08-08 11:00:57.854453] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14560, time_dist: do_cluster_heartbeat=6, generate_cluster_version=14549) [2022-08-08 11:00:57.866117] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=908] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:57.866177] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667656, retry interval=1000000, ret=-4179) [2022-08-08 11:00:58.082460] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:58.086474] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:58.086499] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:58.358373] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:58.358433] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667661, retry interval=1000000, ret=-4179) [2022-08-08 11:00:58.447979] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11135) [2022-08-08 11:00:58.448171] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12706, time_dist: generate_cluster_version=12703) [2022-08-08 11:00:58.584442] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10131) [2022-08-08 11:00:58.585209] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12076, time_dist: do_cluster_heartbeat=46, generate_cluster_version=12026) [2022-08-08 11:00:58.644917] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12967) [2022-08-08 11:00:58.645410] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14260, time_dist: generate_cluster_version=14258) [2022-08-08 11:00:58.899456] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:58.899496] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667655, retry interval=1000000, ret=-4179) [2022-08-08 11:00:58.991194] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=1338] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:59.067229] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13656) [2022-08-08 11:00:59.067472] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16672, time_dist: generate_cluster_version=16670) [2022-08-08 11:00:59.068502] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16259, time_dist: generate_cluster_version=16256) [2022-08-08 11:00:59.089623] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:00:59.093190] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:00:59.093228] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:00:59.236676] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927659236669, cur_ts_=1659927659215141, delta=21528) [2022-08-08 11:00:59.400450] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:59.400484] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667660, retry interval=1000000, ret=-4179) [2022-08-08 11:00:59.401376] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10343, time_dist: do_cluster_heartbeat=12, generate_cluster_version=10328) [2022-08-08 11:00:59.402633] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=27] [dc=0] [trans sampling] (TRACE=begin_ts=1659927659401092 2022-08-08 03:00:59.401092|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:9383744140109628191, inc:84102801, addr:{ip:"192.168.11.128", port:2882}, t:1659927659400000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=85 ret:0, left_time:10000000, uref:1073741825|[start_task] u=8 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7E67-0-0, start_time:1659927659400934, snapshot:1659927659377757|[end_task] u=107 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927659401298|[calc_checksum_by_commit] u=16 ret:0, checksum:2724914733, checksum_log_ts:9223372036854775807|[submit_log] u=55 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1181 trans_version:1659927659400371, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:988208, logging:0, pending:0, uref:1|[end_trans_cb] u=24 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233102640|total_timeu=1502) [2022-08-08 11:00:59.574781] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927659574771, cur_ts_=1659927659550391, delta=24380) [2022-08-08 11:00:59.779983] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:00:59.780051] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:00:59.813747] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=10105) [2022-08-08 11:00:59.814343] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] clock out of order(cur_ts=1659927659814337, cur_ts_=1659927659786104, delta=28233) [2022-08-08 11:00:59.815069] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=603] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12086, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12073) [2022-08-08 11:00:59.942621] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:00:59.942659] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667654, retry interval=1000000, ret=-4179) [2022-08-08 11:01:00.001313] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927660001306, cur_ts_=1659927659978991, delta=22315) [2022-08-08 11:01:00.121382] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:00.122594] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927660122590, cur_ts_=1659927660089336, delta=33254) [2022-08-08 11:01:00.126183] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:00.126239] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:00.250837] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14911744}) [2022-08-08 11:01:00.265892] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16219) [2022-08-08 11:01:00.266221] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30597, time_dist: do_cluster_heartbeat=11, generate_cluster_version=30584) [2022-08-08 11:01:00.501106] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927660501100, cur_ts_=1659927660438285, delta=62815) [2022-08-08 11:01:00.608963] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=340] [dc=0] clock out of order(cur_ts=1659927660608953, cur_ts_=1659927660580652, delta=28301) [2022-08-08 11:01:00.641129] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18809, time_dist: do_cluster_heartbeat=11, generate_cluster_version=18793) [2022-08-08 11:01:00.644783] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927658346644, end_time=1659927660644769, elapsed_time=2298125, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:01:00.659578] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:00.659727] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] task process failed, start retry(max retry time=9223372036854667659, retry interval=1000000, ret=-4179) [2022-08-08 11:01:00.736264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31BB-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:00.736366] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31BB-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:00.736381] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31BB-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:00.737069] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:00.920857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:00.956823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.109343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=2721] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.130037] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927661130027, cur_ts_=1659927661070327, delta=59700) [2022-08-08 11:01:01.148781] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:01.213611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:01.224718] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:01.224754] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:01.254210] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=123281) [2022-08-08 11:01:01.254266] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] slow flush(flush_time=19, io_time=123323, file_id=237, flush_start_offset=52700732, file_offset=52701354) [2022-08-08 11:01:01.254784] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:01.254843] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:01.255654] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659927661255649, cur_ts_=1659927661221632, delta=34017) [2022-08-08 11:01:01.260267] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:13980735639682660602, inc:84102860, addr:"192.168.11.128:2882", t:1659927661070327}, log_id=8302116, used_time=138655) [2022-08-08 11:01:01.260477] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=141150, time_dist: do_cluster_heartbeat=8, generate_cluster_version=141139) [2022-08-08 11:01:01.264868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.283460] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29175) [2022-08-08 11:01:01.285296] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:16527794625873725958, inc:84102862, addr:"192.168.11.128:2882", t:1659927661070327}, log_id=8316710, used_time=162312) [2022-08-08 11:01:01.288624] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=166298, time_dist: do_cluster_heartbeat=7, generate_cluster_version=166286) [2022-08-08 11:01:01.315649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.331730] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15573, time_dist: do_cluster_heartbeat=12, generate_cluster_version=15556) [2022-08-08 11:01:01.344445] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:01.345626] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=558] [dc=0] task process failed, start retry(max retry time=9223372036854667653, retry interval=1000000, ret=-4179) [2022-08-08 11:01:01.375314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.375460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.385680] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.388216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.406960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:01.469421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.478916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:01.491785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.612422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.643557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.699357] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:01.700153] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=444] [dc=0] task process failed, start retry(max retry time=9223372036854667658, retry interval=1000000, ret=-4179) [2022-08-08 11:01:01.751028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:01.751191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:01.784933] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659927661784925, cur_ts_=1659927661762457, delta=22468) [2022-08-08 11:01:01.811806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:01.833436] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=193] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17033) [2022-08-08 11:01:01.833690] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18033, time_dist: generate_cluster_version=18031) [2022-08-08 11:01:02.238804] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:02.242100] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:02.242179] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:02.339188] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927662339178, cur_ts_=1659927662309661, delta=29517) [2022-08-08 11:01:02.442960] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:02.443023] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667652, retry interval=1000000, ret=-4179) [2022-08-08 11:01:02.444352] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=9] [dc=0] [trans sampling] (TRACE=begin_ts=1659927662443501 2022-08-08 03:01:02.443501|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:10758407515463507003, inc:84102909, addr:{ip:"192.168.11.128", port:2882}, t:1659927662437000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7E6C-0-0, start_time:1659927662443357, snapshot:1659927662355835|[end_task] u=154 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927662443672|[calc_checksum_by_commit] u=17 ret:0, checksum:3268086576, checksum_log_ts:9223372036854775807|[submit_log] u=55 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=533 trans_version:1659927662437144, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:988213, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=814) [2022-08-08 11:01:02.692156] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927662692146, cur_ts_=1659927662666361, delta=25785) [2022-08-08 11:01:02.812632] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927662812622, cur_ts_=1659927662735184, delta=77438) [2022-08-08 11:01:02.812675] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=123792) [2022-08-08 11:01:02.813044] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=25] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:8976633546666144842, inc:84102914, addr:"192.168.11.128:2882", t:1659927662666361}, log_id=8316719, used_time=124812) [2022-08-08 11:01:02.813226] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=127911, time_dist: do_cluster_heartbeat=10, generate_cluster_version=127899) [2022-08-08 11:01:02.815061] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] slow flush(flush_time=2373, io_time=123801, file_id=237, flush_start_offset=52716820, file_offset=52717436) [2022-08-08 11:01:02.851616] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36526) [2022-08-08 11:01:02.851970] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:2536970971830570103, inc:84102915, addr:"192.168.11.128:2882", t:1659927662666361}, log_id=8302125, used_time=162131) [2022-08-08 11:01:02.852411] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:02.857416] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] task process failed, start retry(max retry time=9223372036854667657, retry interval=1000000, ret=-4179) [2022-08-08 11:01:02.857616] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=168693, time_dist: do_cluster_heartbeat=10, generate_cluster_version=168681) [2022-08-08 11:01:02.922946] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927662922937, cur_ts_=1659927662874381, delta=48556) [2022-08-08 11:01:02.927602] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=53201) [2022-08-08 11:01:03.059535] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927663059526, cur_ts_=1659927663003945, delta=55581) [2022-08-08 11:01:03.067299] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=131814) [2022-08-08 11:01:03.068676] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=1184] [dc=0] slow flush(flush_time=178, io_time=133013, file_id=237, flush_start_offset=52718745, file_offset=52719934) [2022-08-08 11:01:03.069323] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=8] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:3296329482468321703, inc:84102929, addr:"192.168.11.128:2882", t:1659927662874381}, log_id=8316720, used_time=167192) [2022-08-08 11:01:03.069444] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:5423710177148169886, inc:84102930, addr:"192.168.11.128:2882", t:1659927662874381}, log_id=8302126, used_time=144859) [2022-08-08 11:01:03.069488] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=195270, time_dist: do_cluster_heartbeat=9, generate_cluster_version=195258) [2022-08-08 11:01:03.095482] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=171710, time_dist: do_cluster_heartbeat=10, generate_cluster_version=171696) [2022-08-08 11:01:03.162005] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32084) [2022-08-08 11:01:03.165625] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=40897, time_dist: do_cluster_heartbeat=11, generate_cluster_version=40881) [2022-08-08 11:01:03.176687] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14610) [2022-08-08 11:01:03.176954] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26442, time_dist: do_cluster_heartbeat=13, generate_cluster_version=26426) [2022-08-08 11:01:03.251713] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:03.255010] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:03.255045] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:03.485100] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=1288] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:03.485165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667651, retry interval=1000000, ret=-4179) [2022-08-08 11:01:03.527967] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10810, time_dist: generate_cluster_version=10806) [2022-08-08 11:01:03.818202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:03.857289] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:01:03.934378] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927663934370, cur_ts_=1659927663905852, delta=28518) [2022-08-08 11:01:03.942388] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:03.943390] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667656, retry interval=1000000, ret=-4179) [2022-08-08 11:01:03.944177] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=53511) [2022-08-08 11:01:03.952587] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=211] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=64394, time_dist: do_cluster_heartbeat=10, generate_cluster_version=64379) [2022-08-08 11:01:03.956456] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10740) [2022-08-08 11:01:04.257554] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:04.261128] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:04.261163] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:04.506550] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:04.506656] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667650, retry interval=1000000, ret=-4179) [2022-08-08 11:01:04.594038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:04.689716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:04.708661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:04.712622] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:04.823255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31C7-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:04.823276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31C7-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:04.823478] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31C7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:04.823607] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31C7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:04.823782] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:04.891109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:04.901620] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:04.904642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:04.946139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A75-0-0] [lt=91] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:04.946175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A75-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:04.946196] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A75-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:04.970014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:04.972118] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:04.972179] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667655, retry interval=1000000, ret=-4179) [2022-08-08 11:01:04.974930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:05.008785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.026159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:05.034675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:05.038429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.057417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.060852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.081683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=191] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.109574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:05.111842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.125711] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.126744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:05.126838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.158425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:05.179215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:05.199992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:05.200867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.200920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:05.200958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.200978] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.217142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.234751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.235035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.235361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:05.244832] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.263907] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:05.267545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:05.267651] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.267925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.268058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.268210] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927665268206, cur_ts_=1659927665247706, delta=20500) [2022-08-08 11:01:05.278935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.305811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.324818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:05.332044] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:05.332427] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=367] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:05.324818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=4169] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.364387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.366646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:05.406489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:05.406790] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] clock out of order(cur_ts=1659927665406785, cur_ts_=1659927665382655, delta=24130) [2022-08-08 11:01:05.409490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:05.419499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:05.420001] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42581) [2022-08-08 11:01:05.421044] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45144, time_dist: do_cluster_heartbeat=12, generate_cluster_version=45130) [2022-08-08 11:01:05.422419] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=45678, time_dist: do_cluster_heartbeat=7, generate_cluster_version=45668) [2022-08-08 11:01:05.433333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.511352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.511447] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:05.547857] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927665547831, cur_ts_=1659927665511583, delta=36248) [2022-08-08 11:01:05.548391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.557087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.653160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:05.672089] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=190] [dc=0] clock out of order(cur_ts=1659927665672078, cur_ts_=1659927665511583, delta=160495) [2022-08-08 11:01:05.697799] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=144165) [2022-08-08 11:01:05.714010] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=16168] [dc=0] slow flush(flush_time=14, io_time=160365, file_id=237, flush_start_offset=52741658, file_offset=52742280) [2022-08-08 11:01:05.714939] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=8] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:12067182035018211756, inc:84103027, addr:"192.168.11.128:2882", t:1659927665511583}, log_id=8302140, used_time=162437) [2022-08-08 11:01:05.726888] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=179449, time_dist: do_cluster_heartbeat=10, generate_cluster_version=179433) [2022-08-08 11:01:05.712057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.732355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:05.738640] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=822] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23784) [2022-08-08 11:01:05.738697] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=10] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:3627556546288658695, inc:84103028, addr:"192.168.11.128:2882", t:1659927665511583}, log_id=8316734, used_time=183035) [2022-08-08 11:01:05.741409] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=188789, time_dist: do_cluster_heartbeat=5, generate_cluster_version=188728) [2022-08-08 11:01:05.791503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:05.828044] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE31CA-0-0] [lt=24] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927665825805 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927665827162 abs_expired_time=1659927667757224 stmt_timeout=1930062 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21594983 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927665713479 start_version=-1 trans_version=1659927665825795 commit_version=0 stmt_start_time=1659927665756729 abs_expired_time=1659927695671642 stmt_timeout=29914913 abs_lock_wait_timeout=1659927695671642 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:1372034491053838839, inc:84103032, addr:"192.168.11.128:2882", t:1659927665756729} checksum_log_ts=9223372036854775807, version=1659927665825805) [2022-08-08 11:01:05.834710] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=590] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:05.835060] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667649, retry interval=1000000, ret=-4179) [2022-08-08 11:01:05.867528] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13538) [2022-08-08 11:01:05.872125] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19300, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19285) [2022-08-08 11:01:05.997915] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:05.997950] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667654, retry interval=1000000, ret=-4179) [2022-08-08 11:01:06.008362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:06.390722] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=82963) [2022-08-08 11:01:06.391125] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=85154, time_dist: generate_cluster_version=85150) [2022-08-08 11:01:06.394535] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38341, time_dist: do_cluster_heartbeat=14, generate_cluster_version=38324) [2022-08-08 11:01:06.402600] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:06.408046] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:06.408074] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:06.572963] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=41764] [dc=0] clock out of order(cur_ts=1659927666572957, cur_ts_=1659927666551495, delta=21462) [2022-08-08 11:01:06.781163] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927666781156, cur_ts_=1659927666752569, delta=28587) [2022-08-08 11:01:06.783357] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32412) [2022-08-08 11:01:06.783730] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33525, time_dist: generate_cluster_version=33523) [2022-08-08 11:01:06.958360] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:06.958689] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667648, retry interval=1000000, ret=-4179) [2022-08-08 11:01:07.026733] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:07.026804] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667653, retry interval=1000000, ret=-4179) [2022-08-08 11:01:07.307289] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927667307279, cur_ts_=1659927667261451, delta=45828) [2022-08-08 11:01:07.410342] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:07.413600] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:07.413645] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:07.414646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:07.883072] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927667882763, cur_ts_=1659927667861034, delta=21729) [2022-08-08 11:01:07.989915] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:07.989970] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667647, retry interval=1000000, ret=-4179) [2022-08-08 11:01:07.990007] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11117, time_dist: do_cluster_heartbeat=7, generate_cluster_version=11108) [2022-08-08 11:01:08.059456] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:08.059608] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] task process failed, start retry(max retry time=9223372036854667652, retry interval=1000000, ret=-4179) [2022-08-08 11:01:08.418683] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:08.422255] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:08.422280] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:08.519439] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927668519434, cur_ts_=1659927668492730, delta=26704) [2022-08-08 11:01:08.658050] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19413) [2022-08-08 11:01:08.658720] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21075, time_dist: do_cluster_heartbeat=24, generate_cluster_version=21048) [2022-08-08 11:01:08.681282] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23183) [2022-08-08 11:01:08.682200] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=33234, time_dist: do_cluster_heartbeat=11, generate_cluster_version=33219) [2022-08-08 11:01:08.992566] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6243-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:08.992743] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6243-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:08.992783] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6243-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:08.993478] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6243-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:08.994128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6243-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:08.994156] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6243-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:09.012953] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:09.012993] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667646, retry interval=1000000, ret=-4179) [2022-08-08 11:01:09.048177] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17285) [2022-08-08 11:01:09.048441] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23047, time_dist: generate_cluster_version=23045) [2022-08-08 11:01:09.064988] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=224] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:09.095929] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19880) [2022-08-08 11:01:09.106132] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:09.106170] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667651, retry interval=1000000, ret=-4179) [2022-08-08 11:01:09.341645] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927669341634, cur_ts_=1659927669306034, delta=35600) [2022-08-08 11:01:09.424978] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:09.427971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:09.428008] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:09.593679] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927669593670, cur_ts_=1659927669568209, delta=25461) [2022-08-08 11:01:09.780400] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:01:09.780431] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:01:09.865843] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927669865838, cur_ts_=1659927669838046, delta=27792) [2022-08-08 11:01:09.867845] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23491, time_dist: generate_cluster_version=23487) [2022-08-08 11:01:10.042386] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:10.042439] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667645, retry interval=1000000, ret=-4179) [2022-08-08 11:01:10.141467] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:10.141510] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667650, retry interval=1000000, ret=-4179) [2022-08-08 11:01:10.200843] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927670200834, cur_ts_=1659927670172223, delta=28611) [2022-08-08 11:01:10.203958] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14916224}) [2022-08-08 11:01:10.430295] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:10.434248] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:10.434367] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:10.448671] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927668347251, end_time=1659927670448656, elapsed_time=2101405, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:01:10.449743] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15900, time_dist: generate_cluster_version=15896) [2022-08-08 11:01:10.938915] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:10.977648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.077373] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:11.077460] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667644, retry interval=1000000, ret=-4179) [2022-08-08 11:01:11.127715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=226] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.170201] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:11.170352] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] task process failed, start retry(max retry time=9223372036854667649, retry interval=1000000, ret=-4179) [2022-08-08 11:01:11.234035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:11.286279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.338096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.391712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.391843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.404469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.406399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.422424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:11.436666] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:11.439673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:11.439707] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:11.495492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.497738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:11.515411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.596135] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=49] [dc=0] [trans sampling] (TRACE=begin_ts=1659927671595260 2022-08-08 03:01:11.595260|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:14556729794345348501, inc:84103257, addr:{ip:"192.168.11.128", port:2882}, t:1659927671594000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A64FC-0-0, start_time:1659927671595137, snapshot:1659927671580514|[end_task] u=126 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927671595404|[calc_checksum_by_commit] u=21 ret:0, checksum:1408564536, checksum_log_ts:9223372036854775807|[submit_log] u=79 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=546 trans_version:1659927671595426, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8302170, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=838) [2022-08-08 11:01:11.628562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.665185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.741598] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] clock out of order(cur_ts=1659927671741593, cur_ts_=1659927671714586, delta=27007) [2022-08-08 11:01:11.772249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:11.772503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:11.848756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:12.111222] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:12.111284] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667643, retry interval=1000000, ret=-4179) [2022-08-08 11:01:12.192325] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=178] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:12.192423] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667648, retry interval=1000000, ret=-4179) [2022-08-08 11:01:12.442208] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:12.445514] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:12.445567] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:12.687284] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:13.004783] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31DF-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:13.004895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31DF-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:13.006038] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31DF-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:13.139896] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:13.139944] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667642, retry interval=1000000, ret=-4179) [2022-08-08 11:01:13.215585] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:13.215650] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667647, retry interval=1000000, ret=-4179) [2022-08-08 11:01:13.341273] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] clock out of order(cur_ts=1659927673341263, cur_ts_=1659927673238662, delta=102601) [2022-08-08 11:01:13.365551] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24324) [2022-08-08 11:01:13.366448] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=100442, time_dist: generate_cluster_version=100439) [2022-08-08 11:01:13.378723] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13123) [2022-08-08 11:01:13.380326] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=68914, time_dist: do_cluster_heartbeat=13, generate_cluster_version=68896) [2022-08-08 11:01:13.450109] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:13.453673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:13.453709] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:13.704685] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927673704642, cur_ts_=1659927673675687, delta=28955) [2022-08-08 11:01:14.160531] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:14.160566] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667641, retry interval=1000000, ret=-4179) [2022-08-08 11:01:14.238030] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] clock out of order(cur_ts=1659927674238023, cur_ts_=1659927674197761, delta=40262) [2022-08-08 11:01:14.253386] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16572) [2022-08-08 11:01:14.253648] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17600, time_dist: do_cluster_heartbeat=7, generate_cluster_version=17590) [2022-08-08 11:01:14.294068] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:14.294832] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667646, retry interval=1000000, ret=-4179) [2022-08-08 11:01:14.456030] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:14.459266] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:14.459322] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:14.617970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:14.626461] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927674625635 2022-08-08 03:01:14.625635|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:7661053159176484750, inc:84103372, addr:{ip:"192.168.11.128", port:2882}, t:1659927674625000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC2DC-0-0, start_time:1659927674625489, snapshot:1659927674539360|[end_task] u=126 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927674625777|[calc_checksum_by_commit] u=18 ret:0, checksum:1954502612, checksum_log_ts:9223372036854775807|[submit_log] u=74 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=518 trans_version:1659927674625581, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=1 ret:0, log_type:64, log_id:8316781, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=787) [2022-08-08 11:01:14.705948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:14.728847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:14.729133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:14.913833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:14.918774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:14.928937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:14.994567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:14.994696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:15.025276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.049808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:15.056256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:15.063696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.077573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.126363] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927675126354, cur_ts_=1659927675078702, delta=47652) [2022-08-08 11:01:15.131662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.131688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.131831] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.134613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:15.142802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.142852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.142926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:15.180290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:15.189302] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:15.189411] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] task process failed, start retry(max retry time=9223372036854667640, retry interval=1000000, ret=-4179) [2022-08-08 11:01:15.198406] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:15.223499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:15.223597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.224261] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:15.224332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.231633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.239303] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.278358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.278539] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.278686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.278715] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:15.278774] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927675278771, cur_ts_=1659927675247380, delta=31391) [2022-08-08 11:01:15.282840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:15.288874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.288900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.289258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.299615] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.316577] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:15.316722] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667645, retry interval=1000000, ret=-4179) [2022-08-08 11:01:15.324499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=211] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.343039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27525] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.347333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:15.384590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.385062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:15.431786] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:15.433595] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:15.442266] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:15.448999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.463861] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:15.467162] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:15.467256] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:15.527518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.528154] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:15.569542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.574056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.674573] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=60274] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:15.729839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=17895] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:15.750533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=1277] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:15.807030] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:16.031298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:16.056008] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11774) [2022-08-08 11:01:16.056436] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13493, time_dist: do_cluster_heartbeat=151, generate_cluster_version=13339) [2022-08-08 11:01:16.061526] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:01:16.219975] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:16.220083] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] task process failed, start retry(max retry time=9223372036854667639, retry interval=1000000, ret=-4179) [2022-08-08 11:01:16.302677] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] clock out of order(cur_ts=1659927676302667, cur_ts_=1659927676262870, delta=39797) [2022-08-08 11:01:16.332219] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:16.332496] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=207] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:16.360520] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:16.360564] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667644, retry interval=1000000, ret=-4179) [2022-08-08 11:01:16.457423] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18228) [2022-08-08 11:01:16.457665] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30074, time_dist: do_cluster_heartbeat=13, generate_cluster_version=30059) [2022-08-08 11:01:16.458221] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24179, time_dist: do_cluster_heartbeat=9, generate_cluster_version=24168) [2022-08-08 11:01:16.484006] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:16.485525] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927676485520, cur_ts_=1659927676458248, delta=27272) [2022-08-08 11:01:16.491991] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:16.492014] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:17.002775] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927677002764, cur_ts_=1659927676980981, delta=21783) [2022-08-08 11:01:17.071860] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13262) [2022-08-08 11:01:17.072550] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14746, time_dist: generate_cluster_version=14742) [2022-08-08 11:01:17.245632] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:17.245721] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667638, retry interval=1000000, ret=-4179) [2022-08-08 11:01:17.388427] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:17.388461] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667643, retry interval=1000000, ret=-4179) [2022-08-08 11:01:17.440321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:17.499766] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:17.503091] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:17.503160] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:17.620605] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927677620599, cur_ts_=1659927677595518, delta=25081) [2022-08-08 11:01:17.675929] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] [trans sampling] (TRACE=begin_ts=1659927677674517 2022-08-08 03:01:17.674517|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17174451028035172464, inc:84103479, addr:{ip:"192.168.11.128", port:2882}, t:1659927677671000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A651B-0-0, start_time:1659927677674398, snapshot:1659927677658666|[end_task] u=146 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927677674681|[calc_checksum_by_commit] u=20 ret:0, checksum:1511204741, checksum_log_ts:9223372036854775807|[submit_log] u=27 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1120 trans_version:1659927677674702, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8302201, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1370) [2022-08-08 11:01:17.768238] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2210] [dc=0] clock out of order(cur_ts=1659927677768226, cur_ts_=1659927677722774, delta=45452) [2022-08-08 11:01:17.910207] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2672] [dc=0] clock out of order(cur_ts=1659927677910202, cur_ts_=1659927677888136, delta=22066) [2022-08-08 11:01:18.032232] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12623, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12608) [2022-08-08 11:01:18.269518] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:18.269562] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667637, retry interval=1000000, ret=-4179) [2022-08-08 11:01:18.417056] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:18.417097] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667642, retry interval=1000000, ret=-4179) [2022-08-08 11:01:18.505708] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:18.509626] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:18.509673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:18.604574] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] clock out of order(cur_ts=1659927678604568, cur_ts_=1659927678571105, delta=33463) [2022-08-08 11:01:18.845673] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=13789) [2022-08-08 11:01:18.846741] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927678846733, cur_ts_=1659927678826591, delta=20142) [2022-08-08 11:01:18.848741] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17549, time_dist: do_cluster_heartbeat=5, generate_cluster_version=17540) [2022-08-08 11:01:19.111864] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10657, time_dist: generate_cluster_version=10653) [2022-08-08 11:01:19.191745] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:19.213820] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10590][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=16282) [2022-08-08 11:01:19.292359] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:19.292394] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667636, retry interval=1000000, ret=-4179) [2022-08-08 11:01:19.293651] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11738, time_dist: generate_cluster_version=11736) [2022-08-08 11:01:19.438923] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:19.439133] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667641, retry interval=1000000, ret=-4179) [2022-08-08 11:01:19.512077] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:19.522110] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:19.522152] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:19.783585] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:01:19.783629] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:01:19.983158] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14920640}) [2022-08-08 11:01:20.067129] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31F4-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:20.067508] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31F4-0-0] [lt=347] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:20.067541] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31F4-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:20.067551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE31F4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:20.067831] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:20.167904] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927678347473, end_time=1659927680167891, elapsed_time=1820418, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:01:20.359652] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:20.359863] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927680359858, cur_ts_=1659927680329516, delta=30342) [2022-08-08 11:01:20.359976] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667635, retry interval=1000000, ret=-4179) [2022-08-08 11:01:20.475344] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:20.475379] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667640, retry interval=1000000, ret=-4179) [2022-08-08 11:01:20.527454] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:20.533499] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:20.533569] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:20.690361] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=5] [dc=0] [trans sampling] (TRACE=begin_ts=1659927680689565 2022-08-08 03:01:20.689565|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:9992235441113066890, inc:84103603, addr:{ip:"192.168.11.128", port:2882}, t:1659927680688000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC2FA-0-0, start_time:1659927680689420, snapshot:1659927680644603|[end_task] u=127 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927680689709|[calc_checksum_by_commit] u=18 ret:0, checksum:3593971770, checksum_log_ts:9223372036854775807|[submit_log] u=73 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=489 trans_version:1659927680688167, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8316811, logging:0, pending:0, uref:1|[end_trans_cb] u=15 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=758) [2022-08-08 11:01:20.949844] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927680949839, cur_ts_=1659927680914288, delta=35551) [2022-08-08 11:01:20.960482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.001588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.145933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.254291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:21.312151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.354709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.395849] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:21.395939] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667634, retry interval=1000000, ret=-4179) [2022-08-08 11:01:21.407350] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=215] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.412742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.423616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.431129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.441577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:21.502158] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927681502150, cur_ts_=1659927681470082, delta=32068) [2022-08-08 11:01:21.520525] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:21.520839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.530647] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.580455] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:21.620439] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927681620428, cur_ts_=1659927681470082, delta=150346) [2022-08-08 11:01:21.629755] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=140542) [2022-08-08 11:01:21.629797] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10620][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] slow flush(flush_time=14, io_time=140572, file_id=237, flush_start_offset=52893597, file_offset=52894213) [2022-08-08 11:01:21.630126] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=22] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:993032747919646764, inc:84103629, addr:"192.168.11.128:2882", t:1659927681470082}, log_id=8316815, used_time=140950) [2022-08-08 11:01:21.630481] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=142102, time_dist: generate_cluster_version=142100) [2022-08-08 11:01:21.630670] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:21.630729] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667639, retry interval=1000000, ret=-4179) [2022-08-08 11:01:21.640410] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12228, time_dist: generate_cluster_version=12223) [2022-08-08 11:01:21.641102] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:21.641145] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:21.650149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=498] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.688677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.794849] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=3069] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:21.794897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=3308] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:21.868221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:22.419879] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:22.419927] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667633, retry interval=1000000, ret=-4179) [2022-08-08 11:01:22.450663] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16822) [2022-08-08 11:01:22.451657] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19849, time_dist: do_cluster_heartbeat=12, generate_cluster_version=19827) [2022-08-08 11:01:22.469621] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18623) [2022-08-08 11:01:22.470730] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37791, time_dist: do_cluster_heartbeat=9, generate_cluster_version=37777) [2022-08-08 11:01:22.517873] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927682517868, cur_ts_=1659927682493917, delta=23951) [2022-08-08 11:01:22.644708] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:22.658330] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:22.658358] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:22.666507] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=1818] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:22.666578] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667638, retry interval=1000000, ret=-4179) [2022-08-08 11:01:22.742143] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927682742132, cur_ts_=1659927682710312, delta=31820) [2022-08-08 11:01:23.443113] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:23.443165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667632, retry interval=1000000, ret=-4179) [2022-08-08 11:01:23.661599] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:23.666933] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:23.669320] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=2372] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:23.688298] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:23.688347] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667637, retry interval=1000000, ret=-4179) [2022-08-08 11:01:23.703784] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927683702848 2022-08-08 03:01:23.702848|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:13863782805480318020, inc:84103715, addr:{ip:"192.168.11.128", port:2882}, t:1659927683701000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6539-0-0, start_time:1659927683702692, snapshot:1659927683627947|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927683702990|[calc_checksum_by_commit] u=20 ret:0, checksum:1802561852, checksum_log_ts:9223372036854775807|[submit_log] u=64 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=577 trans_version:1659927683703011, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8302231, logging:0, pending:0, uref:1|[end_trans_cb] u=31 ret:0, param:0, param:true|[destroy] u=43 arg1:140085232375600|total_timeu=895) [2022-08-08 11:01:23.870022] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927683869767, cur_ts_=1659927683841470, delta=28297) [2022-08-08 11:01:23.955006] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=281] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32641) [2022-08-08 11:01:23.955844] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34709, time_dist: do_cluster_heartbeat=14, generate_cluster_version=34692) [2022-08-08 11:01:23.956449] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31279, time_dist: do_cluster_heartbeat=8, generate_cluster_version=31270) [2022-08-08 11:01:24.042712] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927684042706, cur_ts_=1659927684010762, delta=31944) [2022-08-08 11:01:24.124632] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE3200-0-0] [lt=20] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927684122741 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927684124091 abs_expired_time=1659927686053998 stmt_timeout=1929907 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21595243 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927684065101 start_version=-1 trans_version=1659927684109972 commit_version=0 stmt_start_time=1659927684103809 abs_expired_time=1659927714004412 stmt_timeout=29900603 abs_lock_wait_timeout=1659927714004412 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:7643183679731242156, inc:84103825, addr:"192.168.11.128:2882", t:1659927684103809} checksum_log_ts=9223372036854775807, version=1659927684122741) [2022-08-08 11:01:24.299682] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15033, time_dist: generate_cluster_version=15031) [2022-08-08 11:01:24.445396] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.445431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.445659] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.445675] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.445692] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.445704] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446024] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446042] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446417] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446430] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.446439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A90-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:24.472176] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:24.472210] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667631, retry interval=1000000, ret=-4179) [2022-08-08 11:01:24.615328] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927684615323, cur_ts_=1659927684594293, delta=21030) [2022-08-08 11:01:24.636809] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:24.676694] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:24.684451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:24.684482] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:24.707396] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14475, time_dist: generate_cluster_version=14468) [2022-08-08 11:01:24.723120] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:24.761896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:24.762416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:24.763944] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:24.764150] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] task process failed, start retry(max retry time=9223372036854667636, retry interval=1000000, ret=-4179) [2022-08-08 11:01:24.892182] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14317) [2022-08-08 11:01:24.892548] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16522, time_dist: generate_cluster_version=16520) [2022-08-08 11:01:24.903240] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10941) [2022-08-08 11:01:24.903682] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27360, time_dist: do_cluster_heartbeat=6, generate_cluster_version=27352) [2022-08-08 11:01:24.931428] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:24.935322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:24.945619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.011691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:25.013846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.045365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.071851] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:25.083440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:25.089469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.100136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.145704] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3203-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:25.145904] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:25.150952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.151467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.153492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:25.154382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.161685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:25.163589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.174069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.202268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=620] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:25.216441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:25.244882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.245556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.250575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.250631] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.257758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=266] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.260866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.295521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.295466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.299495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.301988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.303283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.306392] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.312364] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.313329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.324240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.348805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.360664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.382657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.382763] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927685382759, cur_ts_=1659927685362021, delta=20738) [2022-08-08 11:01:25.405158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.405528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:25.448634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:25.458799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:25.465130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:25.465236] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.507195] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17607) [2022-08-08 11:01:25.507527] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19111, time_dist: do_cluster_heartbeat=14, generate_cluster_version=19095) [2022-08-08 11:01:25.509404] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.513211] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:25.513295] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667630, retry interval=1000000, ret=-4179) [2022-08-08 11:01:25.544048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=177] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.548173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:25.593184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.595508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.695932] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:25.699198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:25.699205] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:25.699223] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:25.749889] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:25.767068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A93-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.767275] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A93-0-0] [lt=81] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.767409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A93-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.767835] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A93-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.767857] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4A93-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.776377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:25.799861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.799903] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.800062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:25.801664] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:25.806658] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667635, retry interval=1000000, ret=-4179) [2022-08-08 11:01:25.827213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:26.051907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:26.543040] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:26.543103] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667629, retry interval=1000000, ret=-4179) [2022-08-08 11:01:26.701810] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:26.704953] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:26.704988] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:26.715406] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927686713221 2022-08-08 03:01:26.713221|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:16026919522760834604, inc:84104602, addr:{ip:"192.168.11.128", port:2882}, t:1659927686711000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=9 ret:0, left_time:10000000, uref:1073741825|[start_task] u=8 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6548-0-0, start_time:1659927686713023, snapshot:1659927686643343|[end_task] u=139 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927686713383|[calc_checksum_by_commit] u=25 ret:0, checksum:3738917675, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1880 trans_version:1659927686713410, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8302246, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=2139) [2022-08-08 11:01:26.812774] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:26.812811] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:26.837605] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:26.837750] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:26.837836] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:26.839184] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:26.839235] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667634, retry interval=1000000, ret=-4179) [2022-08-08 11:01:26.852639] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12746) [2022-08-08 11:01:27.172872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3209-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.172923] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3209-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.173155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3209-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.173299] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3209-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.173457] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.173674] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.173820] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.173941] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:27.456638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:27.613935] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:27.614052] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667628, retry interval=1000000, ret=-4179) [2022-08-08 11:01:27.707792] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:27.713860] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:27.713909] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:27.872636] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:27.872658] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:27.872701] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:27.874772] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:27.874858] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667633, retry interval=1000000, ret=-4179) [2022-08-08 11:01:28.659272] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:28.659357] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667627, retry interval=1000000, ret=-4179) [2022-08-08 11:01:28.716671] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:28.720811] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:28.720838] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:28.911837] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:28.911871] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667632, retry interval=1000000, ret=-4179) [2022-08-08 11:01:29.092436] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927689092432, cur_ts_=1659927689056934, delta=35498) [2022-08-08 11:01:29.167683] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15308, time_dist: do_cluster_heartbeat=12, generate_cluster_version=15292) [2022-08-08 11:01:29.168393] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14919, time_dist: do_cluster_heartbeat=6, generate_cluster_version=14910) [2022-08-08 11:01:29.300213] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:29.672493] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:29.672638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:29.692992] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:29.693027] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667626, retry interval=1000000, ret=-4179) [2022-08-08 11:01:29.732463] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:29.735671] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:29.735714] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:29.745534] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927689740750 2022-08-08 03:01:29.740750|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:1209365555538177366, inc:84104723, addr:{ip:"192.168.11.128", port:2882}, t:1659927689739000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6557-0-0, start_time:1659927689740629, snapshot:1659927689689484|[end_task] u=1776 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=13 stc:1659927689742546|[calc_checksum_by_commit] u=415 ret:0, checksum:2519593598, checksum_log_ts:9223372036854775807|[submit_log] u=35 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2446 trans_version:1659927689742576, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8302261, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=4739) [2022-08-08 11:01:29.784381] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:01:29.784414] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:01:29.927222] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15260) [2022-08-08 11:01:29.927520] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16385, time_dist: generate_cluster_version=16382) [2022-08-08 11:01:29.969137] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:29.969179] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667631, retry interval=1000000, ret=-4179) [2022-08-08 11:01:30.161636] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15190) [2022-08-08 11:01:30.162871] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14924960}) [2022-08-08 11:01:30.172340] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28138, time_dist: generate_cluster_version=28134) [2022-08-08 11:01:30.193979] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE3212-0-0] [lt=16] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927690191878 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927690191868 abs_expired_time=1659927692123342 stmt_timeout=1931474 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21595333 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927690142739 start_version=-1 trans_version=1659927690191868 commit_version=0 stmt_start_time=1659927690165433 abs_expired_time=1659927720067274 stmt_timeout=29901841 abs_lock_wait_timeout=1659927720067274 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:2046349468930027824, inc:84104736, addr:"192.168.11.128:2882", t:1659927690165433} checksum_log_ts=9223372036854775807, version=1659927690191878) [2022-08-08 11:01:30.394381] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927688348609, end_time=1659927690394367, elapsed_time=2045758, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:01:30.699029] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BDB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.699292] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BDB-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.699308] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BDB-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.699502] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BDB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.699515] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4BDB-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.720119] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.720137] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.720198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.721356] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:30.721422] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667625, retry interval=1000000, ret=-4179) [2022-08-08 11:01:30.737376] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:30.740686] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:30.740742] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:30.985065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:30.991555] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.991905] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:30.996374] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:30.996418] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667630, retry interval=1000000, ret=-4179) [2022-08-08 11:01:31.017357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.162960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.276797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:31.328933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.371246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.423370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.432876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.441691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.449904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.464323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:31.544676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:31.544976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=435] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.555592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.672737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.703859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.743757] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:31.748457] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:31.748860] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:31.748952] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:31.750995] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:31.751048] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667624, retry interval=1000000, ret=-4179) [2022-08-08 11:01:31.754102] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=2303] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:31.754216] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:31.818247] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:31.818391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:31.891105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:31.971183] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927691971174, cur_ts_=1659927691949852, delta=21322) [2022-08-08 11:01:31.971766] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26109, time_dist: generate_cluster_version=26106) [2022-08-08 11:01:32.019845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:32.019865] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:32.021155] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:32.021204] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667629, retry interval=1000000, ret=-4179) [2022-08-08 11:01:32.206783] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3218-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.206929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3218-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.207109] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3218-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.207197] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3218-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.207306] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.207431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.207582] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.207724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:32.687788] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:32.761462] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:32.783189] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927692783179, cur_ts_=1659927692763080, delta=20099) [2022-08-08 11:01:32.783762] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21645) [2022-08-08 11:01:32.788502] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:32.788530] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:32.789910] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32017, time_dist: generate_cluster_version=32013) [2022-08-08 11:01:32.847185] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:32.847257] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667623, retry interval=1000000, ret=-4179) [2022-08-08 11:01:33.044787] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:33.044806] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:33.044863] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:33.046486] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:33.046537] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667628, retry interval=1000000, ret=-4179) [2022-08-08 11:01:33.797243] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:33.800380] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:33.800444] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:33.875729] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:33.876042] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:33.878751] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:33.878934] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667622, retry interval=1000000, ret=-4179) [2022-08-08 11:01:34.070792] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:34.070856] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667627, retry interval=1000000, ret=-4179) [2022-08-08 11:01:34.654404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:34.747567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:34.778017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:34.779873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:34.803112] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:34.808779] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:34.808892] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:34.903908] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:34.903969] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667621, retry interval=1000000, ret=-4179) [2022-08-08 11:01:34.956399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:34.956529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:34.964055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.030385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:35.039018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.063551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=1876] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.116496] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:35.118454] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:35.118568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.119061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=273] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.120645] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927695120640, cur_ts_=1659927695085554, delta=35086) [2022-08-08 11:01:35.126047] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:35.126084] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667626, retry interval=1000000, ret=-4179) [2022-08-08 11:01:35.170224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.172148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.172252] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.172276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:35.182646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:35.184512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.191705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.222339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:35.232672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=545] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:35.236884] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3221-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.236971] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3221-0-0] [lt=63] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.237191] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3221-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.237318] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3221-0-0] [lt=109] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.237407] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.237444] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.237661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.237766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:35.264228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.265555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.276933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.277762] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.279908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.280125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.315032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.317514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.317667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=257] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.321401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.325068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.326974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=1839] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.335451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.338534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.350409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.373155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.377948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.402553] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.423247] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.431076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:35.467088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:35.476872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:35.486066] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.489864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:35.562531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.574603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:35.615908] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.617708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.723791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:35.780095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.802050] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:35.811625] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:35.816042] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:35.816080] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:35.842864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:35.907592] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A65-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:35.933906] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:35.934095] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] task process failed, start retry(max retry time=9223372036854667620, retry interval=1000000, ret=-4179) [2022-08-08 11:01:36.075461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:36.091556] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:01:36.150395] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:36.150476] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667625, retry interval=1000000, ret=-4179) [2022-08-08 11:01:36.818496] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:36.821485] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:36.821520] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:36.936704] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4AA2-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:36.936750] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4AA2-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:36.936762] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4AA2-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:36.955021] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:36.956205] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:36.956252] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667619, retry interval=1000000, ret=-4179) [2022-08-08 11:01:37.155486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:37.173568] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:37.175244] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:37.175288] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667624, retry interval=1000000, ret=-4179) [2022-08-08 11:01:37.253186] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3227-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:37.253392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3227-0-0] [lt=159] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:37.253489] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3227-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:37.253666] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3227-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:37.253795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:37.480693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=830] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:37.784465] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927697784460, cur_ts_=1659927697753682, delta=30778) [2022-08-08 11:01:37.846145] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=32421) [2022-08-08 11:01:37.846361] WARN [STORAGE.TRANS] submit_log_impl_ (ob_trans_part_ctx.cpp:6697) [11016][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] submitting log costs too much time(ret=0, ret="OB_SUCCESS", timeguard=time guard 'submit_log_impl' cost too much time, used=32673, time_dist: 1,27,32640,2, trans_id={hash:10803186203692274737, inc:84105021, addr:"192.168.11.128:2882", t:1659927697809604}, partition={tid:1100611139404002, partition_id:0, part_cnt:0}, state={prepare_version:1659927697813696, state:0}, real_log_type=64, log_size=469) [2022-08-08 11:01:37.846371] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard 'submit_log_impl' cost too much time, used=32694, time_dist: 1,27,32640,2) [2022-08-08 11:01:37.850303] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:37.861445] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12316) [2022-08-08 11:01:37.861874] TRACE [TRACE]ob_trans_ctx.cpp:487 [10630][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] [trans warn] (TRACE=begin_ts=1659927697812012 2022-08-08 03:01:37.812012|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:10803186203692274737, inc:84105021, addr:{ip:"192.168.11.128", port:2882}, t:1659927697809604}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A657F-0-0, start_time:1659927697811341, snapshot:1659927697752580|[end_task] u=1635 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=13 stc:1659927697813670|[calc_checksum_by_commit] u=25 ret:0, checksum:2374824738, checksum_log_ts:9223372036854775807|[submit_log] u=32645 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=32 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=15413 trans_version:1659927697813696, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8302301, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=49818) [2022-08-08 11:01:37.861924] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=65425, time_dist: do_cluster_heartbeat=9, generate_cluster_version=65413) [2022-08-08 11:01:37.866442] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:37.866471] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:37.897460] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927697897451, cur_ts_=1659927697872291, delta=25160) [2022-08-08 11:01:37.905908] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=44393) [2022-08-08 11:01:37.930881] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=84946, time_dist: print_stat=7, generate_cluster_version=84931, cluster_service_self_check=5) [2022-08-08 11:01:38.027084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.027104] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.027110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.027115] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.027128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.027137] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.027150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.028369] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.028459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.028485] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.028508] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC626D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.037962] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927698037952, cur_ts_=1659927697995613, delta=42339) [2022-08-08 11:01:38.093951] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=101584) [2022-08-08 11:01:38.093982] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=12, io_time=101609, file_id=237, flush_start_offset=53043134, file_offset=53043750) [2022-08-08 11:01:38.094043] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=7] [dc=0] transaction log sync use too much time(partition={tid:1099511628002, partition_id:0, part_cnt:0}, trans_id={hash:1479683202912588945, inc:84105024, addr:"192.168.11.128:2882", t:1659927697981331}, log_id=8316897, used_time=110036) [2022-08-08 11:01:38.094224] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=110880, time_dist: do_cluster_heartbeat=8, generate_cluster_version=110867) [2022-08-08 11:01:38.203717] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927698203709, cur_ts_=1659927698172641, delta=31068) [2022-08-08 11:01:38.207079] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=48109) [2022-08-08 11:01:38.209478] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=55642, time_dist: do_cluster_heartbeat=13, generate_cluster_version=55627) [2022-08-08 11:01:38.213591] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=54344, time_dist: do_cluster_heartbeat=11, generate_cluster_version=54328) [2022-08-08 11:01:38.221486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.221524] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.221530] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:38.223445] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:38.224340] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667618, retry interval=1000000, ret=-4179) [2022-08-08 11:01:38.257506] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:38.257542] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667623, retry interval=1000000, ret=-4179) [2022-08-08 11:01:38.258449] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27655) [2022-08-08 11:01:38.268136] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:38.268166] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:38.268178] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:38.268184] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:38.325516] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927698325504, cur_ts_=1659927698298127, delta=27377) [2022-08-08 11:01:38.394937] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11526, time_dist: do_cluster_heartbeat=11, generate_cluster_version=11510) [2022-08-08 11:01:38.801081] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] [trans sampling] (TRACE=begin_ts=1659927698799147 2022-08-08 03:01:38.799147|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:18311388745342185714, inc:84105061, addr:{ip:"192.168.11.128", port:2882}, t:1659927698798000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC354-0-0, start_time:1659927698799021, snapshot:1659927698798945|[end_task] u=133 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927698799298|[calc_checksum_by_commit] u=18 ret:0, checksum:2450283712, checksum_log_ts:9223372036854775807|[submit_log] u=55 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1516 trans_version:1659927698798946, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=4 ret:0, log_type:64, log_id:8316901, logging:0, pending:0, uref:1|[end_trans_cb] u=23 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=1789) [2022-08-08 11:01:38.839837] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=13252) [2022-08-08 11:01:38.842211] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17825, time_dist: generate_cluster_version=17820) [2022-08-08 11:01:38.901915] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:38.905778] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:38.905802] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:39.045606] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43829) [2022-08-08 11:01:39.045827] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=44820, time_dist: generate_cluster_version=44818) [2022-08-08 11:01:39.067624] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21974) [2022-08-08 11:01:39.067876] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=43550, time_dist: generate_cluster_version=43548) [2022-08-08 11:01:39.234462] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12639, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12616) [2022-08-08 11:01:39.291344] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:39.291383] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667622, retry interval=1000000, ret=-4179) [2022-08-08 11:01:39.299293] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:39.299328] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667617, retry interval=1000000, ret=-4179) [2022-08-08 11:01:39.307465] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14275) [2022-08-08 11:01:39.364826] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:39.425944] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=162] [dc=0] clock out of order(cur_ts=1659927699425920, cur_ts_=1659927699395265, delta=30655) [2022-08-08 11:01:39.446401] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19017) [2022-08-08 11:01:39.447271] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8986] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21489, time_dist: do_cluster_heartbeat=8, generate_cluster_version=21469) [2022-08-08 11:01:39.452190] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25509, time_dist: do_cluster_heartbeat=6, generate_cluster_version=25498) [2022-08-08 11:01:39.789150] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:01:39.789168] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:01:39.907637] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:39.915946] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:39.916078] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:40.309695] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10950) [2022-08-08 11:01:40.348485] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:40.348609] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667621, retry interval=1000000, ret=-4179) [2022-08-08 11:01:40.349857] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:40.349888] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667616, retry interval=1000000, ret=-4179) [2022-08-08 11:01:40.362800] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=516] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14929312}) [2022-08-08 11:01:40.371834] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11610, time_dist: do_cluster_heartbeat=5, generate_cluster_version=11601) [2022-08-08 11:01:40.551050] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927698356153, end_time=1659927700551039, elapsed_time=2194886, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:01:40.903825] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927700903805, cur_ts_=1659927700880355, delta=23450) [2022-08-08 11:01:40.930951] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:40.939812] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:40.939853] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:41.002546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.026597] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10675, time_dist: do_cluster_heartbeat=8, generate_cluster_version=10663) [2022-08-08 11:01:41.034029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.146020] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] clock out of order(cur_ts=1659927701146011, cur_ts_=1659927701115410, delta=30601) [2022-08-08 11:01:41.185003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.291946] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:41.344901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.377946] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11996, time_dist: do_cluster_heartbeat=8, generate_cluster_version=11986) [2022-08-08 11:01:41.382893] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:41.382930] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667615, retry interval=1000000, ret=-4179) [2022-08-08 11:01:41.391493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.410405] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=2677] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:41.410739] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] task process failed, start retry(max retry time=9223372036854667620, retry interval=1000000, ret=-4179) [2022-08-08 11:01:41.440776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.451086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.462273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.466049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.487642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:41.562411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:41.568816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.572319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.682036] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927701682030, cur_ts_=1659927701630884, delta=51146) [2022-08-08 11:01:41.701649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.705202] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=45401) [2022-08-08 11:01:41.707625] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=48581, time_dist: generate_cluster_version=48498) [2022-08-08 11:01:41.715454] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34336, time_dist: do_cluster_heartbeat=8, generate_cluster_version=34324) [2022-08-08 11:01:41.727403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.824369] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927701823216 2022-08-08 03:01:41.823216|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:11461093479337202262, inc:84105182, addr:{ip:"192.168.11.128", port:2882}, t:1659927701822000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC364-0-0, start_time:1659927701822902, snapshot:1659927701780575|[end_task] u=154 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927701823388|[calc_checksum_by_commit] u=22 ret:0, checksum:3945059411, checksum_log_ts:9223372036854775807|[submit_log] u=41 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=835 trans_version:1659927701822450, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8316917, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=1110) [2022-08-08 11:01:41.840623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:41.842294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:41.916011] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:41.942327] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:41.950716] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:41.955178] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=4444] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:41.996049] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10932, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10916) [2022-08-08 11:01:42.049177] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659927702049167, cur_ts_=1659927702009191, delta=39976) [2022-08-08 11:01:42.059805] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22099, time_dist: do_cluster_heartbeat=26, generate_cluster_version=22069) [2022-08-08 11:01:42.158480] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927702158470, cur_ts_=1659927702127692, delta=30778) [2022-08-08 11:01:42.341530] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3236-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:42.341573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3236-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:42.341585] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3236-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:42.447712] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:42.447775] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667614, retry interval=1000000, ret=-4179) [2022-08-08 11:01:42.451402] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:42.451444] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:42.453124] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:42.453318] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] task process failed, start retry(max retry time=9223372036854667619, retry interval=1000000, ret=-4179) [2022-08-08 11:01:42.818851] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927702818839, cur_ts_=1659927702798819, delta=20020) [2022-08-08 11:01:42.863950] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27075) [2022-08-08 11:01:42.865434] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30117, time_dist: do_cluster_heartbeat=11, generate_cluster_version=30102) [2022-08-08 11:01:42.973668] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:42.979209] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:42.979247] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:43.327042] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927703327034, cur_ts_=1659927703304353, delta=22681) [2022-08-08 11:01:43.347191] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3239-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.347423] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3239-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.347577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3239-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.347691] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.347878] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.348017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.348137] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:43.446823] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927703446815, cur_ts_=1659927703405412, delta=41403) [2022-08-08 11:01:43.456059] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19816) [2022-08-08 11:01:43.456312] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20779, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20769) [2022-08-08 11:01:43.466576] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10475) [2022-08-08 11:01:43.485009] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37895, time_dist: do_cluster_heartbeat=9, generate_cluster_version=37883) [2022-08-08 11:01:43.525963] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:43.526692] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=557] [dc=0] task process failed, start retry(max retry time=9223372036854667618, retry interval=1000000, ret=-4179) [2022-08-08 11:01:43.528240] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:43.528282] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667613, retry interval=1000000, ret=-4179) [2022-08-08 11:01:43.659440] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21349) [2022-08-08 11:01:43.659665] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22998, time_dist: generate_cluster_version=22995) [2022-08-08 11:01:43.693825] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34318) [2022-08-08 11:01:43.694651] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=963] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=36515, time_dist: generate_cluster_version=36510) [2022-08-08 11:01:43.940834] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=56675) [2022-08-08 11:01:43.941505] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=59114, time_dist: do_cluster_heartbeat=10, generate_cluster_version=59101) [2022-08-08 11:01:43.945654] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=62490, time_dist: do_cluster_heartbeat=4, generate_cluster_version=62482) [2022-08-08 11:01:43.982125] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:43.991334] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:43.991382] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:44.066587] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17504) [2022-08-08 11:01:44.067129] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18822, time_dist: do_cluster_heartbeat=10, generate_cluster_version=18809) [2022-08-08 11:01:44.123152] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927704123141, cur_ts_=1659927704095873, delta=27268) [2022-08-08 11:01:44.354409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:44.354528] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:44.529308] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6276-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:44.529557] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6276-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:44.529668] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6276-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:44.558689] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:44.558725] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667617, retry interval=1000000, ret=-4179) [2022-08-08 11:01:44.561808] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:44.561869] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667612, retry interval=1000000, ret=-4179) [2022-08-08 11:01:44.675957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:44.766569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:44.799191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:44.803125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:44.979928] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:44.979965] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:44.979980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:44.998068] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:45.002719] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:45.002765] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:45.048716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:45.064141] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.081478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.136977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:45.141935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=490] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:45.143981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.144057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.198059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.201756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.198175] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:45.198992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.199056] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927705199053, cur_ts_=1659927705171195, delta=27858) [2022-08-08 11:01:45.209036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.209272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.212040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:45.240302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:45.250694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:45.280804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.281637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.295575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.302134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.302158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.302430] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.333355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.333422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.338179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.344064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.344064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.348243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.352728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.354497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.360068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE323F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:45.360395] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE323F-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:45.360429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE323F-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:45.375158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.390719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.401925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.427419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.448703] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:45.449014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.489866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:45.500343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:45.510548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:45.510811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.588216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.594218] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:45.594283] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667611, retry interval=1000000, ret=-4179) [2022-08-08 11:01:45.596261] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:45.596290] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667616, retry interval=1000000, ret=-4179) [2022-08-08 11:01:45.598939] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:45.631850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.642425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.746923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:45.801866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:45.819417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:45.866337] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:46.005291] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:46.017137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:46.017175] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:46.093046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:46.670730] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17477, time_dist: do_cluster_heartbeat=9, generate_cluster_version=17465) [2022-08-08 11:01:46.672636] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10363, time_dist: generate_cluster_version=10361) [2022-08-08 11:01:46.681883] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:46.681932] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667610, retry interval=1000000, ret=-4179) [2022-08-08 11:01:46.683164] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=451] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:46.683232] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667615, retry interval=1000000, ret=-4179) [2022-08-08 11:01:46.977037] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6968] [dc=0] clock out of order(cur_ts=1659927706977027, cur_ts_=1659927706933800, delta=43227) [2022-08-08 11:01:47.019648] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:47.023323] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:47.023364] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:47.402312] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3245-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:47.402334] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3245-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:47.402452] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:47.402577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:47.402802] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:47.402901] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:47.496890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:47.724755] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=557] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:47.726948] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=2064] [dc=0] task process failed, start retry(max retry time=9223372036854667609, retry interval=1000000, ret=-4179) [2022-08-08 11:01:47.729425] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:47.729460] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667614, retry interval=1000000, ret=-4179) [2022-08-08 11:01:48.027507] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:48.030776] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:48.030822] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:48.203373] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] clock out of order(cur_ts=1659927708203367, cur_ts_=1659927708166692, delta=36675) [2022-08-08 11:01:48.314822] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927708314814, cur_ts_=1659927708292405, delta=22409) [2022-08-08 11:01:48.424346] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3248-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:48.424374] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3248-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:48.424468] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:48.424479] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:48.450288] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13585, time_dist: do_cluster_heartbeat=8, generate_cluster_version=13573) [2022-08-08 11:01:48.607898] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927708607892, cur_ts_=1659927708587479, delta=20413) [2022-08-08 11:01:48.784511] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927708784504, cur_ts_=1659927708758270, delta=26234) [2022-08-08 11:01:48.902167] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=36674) [2022-08-08 11:01:48.903059] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:48.903058] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=38456, time_dist: do_cluster_heartbeat=10, generate_cluster_version=38444) [2022-08-08 11:01:48.904138] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=823] [dc=0] task process failed, start retry(max retry time=9223372036854667613, retry interval=1000000, ret=-4179) [2022-08-08 11:01:48.925005] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24428, time_dist: do_cluster_heartbeat=11, generate_cluster_version=24413) [2022-08-08 11:01:48.931381] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27040) [2022-08-08 11:01:48.956346] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:48.956395] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854667608, retry interval=1000000, ret=-4179) [2022-08-08 11:01:49.021894] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927709021888, cur_ts_=1659927708998619, delta=23269) [2022-08-08 11:01:49.038743] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:49.071279] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:49.071309] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:49.168451] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18442) [2022-08-08 11:01:49.171158] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26325, time_dist: do_cluster_heartbeat=24, generate_cluster_version=26296) [2022-08-08 11:01:49.172529] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25930, time_dist: do_cluster_heartbeat=15, generate_cluster_version=25910) [2022-08-08 11:01:49.465574] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:49.502490] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35051) [2022-08-08 11:01:49.536696] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28992) [2022-08-08 11:01:49.537154] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31182, time_dist: print_stat=6, generate_cluster_version=31172) [2022-08-08 11:01:49.571654] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927709571645, cur_ts_=1659927709547810, delta=23835) [2022-08-08 11:01:49.595406] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=35162) [2022-08-08 11:01:49.616381] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10405) [2022-08-08 11:01:49.790673] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:01:49.790692] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:01:49.892730] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10042) [2022-08-08 11:01:49.893690] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12400, time_dist: generate_cluster_version=12396) [2022-08-08 11:01:49.953139] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:49.953177] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667612, retry interval=1000000, ret=-4179) [2022-08-08 11:01:49.983236] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:49.983785] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667607, retry interval=1000000, ret=-4179) [2022-08-08 11:01:50.093458] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:50.099101] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:50.099144] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:50.630262] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14933632}) [2022-08-08 11:01:50.824166] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927708354062, end_time=1659927710824154, elapsed_time=2470092, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:01:50.914123] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=7] [dc=0] [trans sampling] (TRACE=begin_ts=1659927710908041 2022-08-08 03:01:50.908041|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:10534584626091455175, inc:84105515, addr:{ip:"192.168.11.128", port:2882}, t:1659927710906000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC391-0-0, start_time:1659927710907926, snapshot:1659927710891023|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927710908181|[calc_checksum_by_commit] u=18 ret:0, checksum:198743316, checksum_log_ts:9223372036854775807|[submit_log] u=125 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=5715 trans_version:1659927710906793, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8316962, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=6039) [2022-08-08 11:01:50.983310] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:50.983373] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667611, retry interval=1000000, ret=-4179) [2022-08-08 11:01:51.006364] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:51.006423] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667606, retry interval=1000000, ret=-4179) [2022-08-08 11:01:51.018914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=506] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.051126] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.106261] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:51.109558] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:51.109626] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:51.200251] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.306720] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927711306712, cur_ts_=1659927711284772, delta=21940) [2022-08-08 11:01:51.322027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:51.363633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.411295] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.463546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.473900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.483728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.484268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.505554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:51.583987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:51.590336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.590518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.722253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.745586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.860606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=255] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:51.865149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:51.937044] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927711937031, cur_ts_=1659927711903239, delta=33792) [2022-08-08 11:01:51.937900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:52.009678] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:52.009714] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667610, retry interval=1000000, ret=-4179) [2022-08-08 11:01:52.029817] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:52.029880] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667605, retry interval=1000000, ret=-4179) [2022-08-08 11:01:52.112599] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=162] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:52.115694] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:52.115748] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:52.687997] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:53.019591] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:53.019619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:01:53.058540] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:53.058588] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667604, retry interval=1000000, ret=-4179) [2022-08-08 11:01:53.060821] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:53.060885] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667609, retry interval=1000000, ret=-4179) [2022-08-08 11:01:53.118105] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:53.122205] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:53.122244] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:53.558062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3257-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:53.558383] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3257-0-0] [lt=266] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:53.558403] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3257-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:53.558490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3257-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:54.084493] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:54.084526] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667603, retry interval=1000000, ret=-4179) [2022-08-08 11:01:54.091390] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:54.091421] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667608, retry interval=1000000, ret=-4179) [2022-08-08 11:01:54.124629] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:54.127735] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:54.127779] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:54.234112] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927714234103, cur_ts_=1659927714198217, delta=35886) [2022-08-08 11:01:54.563878] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE325A-0-0] [lt=5] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927714563664 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927714561699 abs_expired_time=1659927716492819 stmt_timeout=1931120 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21595667 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927714511685 start_version=-1 trans_version=1659927714561699 commit_version=0 stmt_start_time=1659927714552209 abs_expired_time=1659927744453896 stmt_timeout=29901687 abs_lock_wait_timeout=1659927744453896 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:3100045392742121110, inc:84105651, addr:"192.168.11.128:2882", t:1659927714552209} checksum_log_ts=9223372036854775807, version=1659927714563664) [2022-08-08 11:01:54.576500] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13169) [2022-08-08 11:01:54.691750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:54.787373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:54.824605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:54.824699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.003232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.004591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:55.010729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.067063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:55.085195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.101667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.122387] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:55.122437] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667607, retry interval=1000000, ret=-4179) [2022-08-08 11:01:55.124141] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11111, time_dist: generate_cluster_version=11109) [2022-08-08 11:01:55.124795] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:55.124863] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667602, retry interval=1000000, ret=-4179) [2022-08-08 11:01:55.130037] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:55.133153] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:55.133189] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:55.157376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:55.163354] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:55.164841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.167720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.216964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=1592] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.217052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=7733] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:55.217830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=7023] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.218548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.227052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.227311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.236873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:55.257554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:01:55.269492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=190] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:55.302308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.302347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.312491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.322601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.322808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.322997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.353852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.353880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.354107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.364257] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.364358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.364371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=156] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.374805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.375196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.396017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.405859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.426691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.447677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.468827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:55.468992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.510012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:01:55.520479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=195] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:01:55.532334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.532380] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:55.604714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.615678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:55.656407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.666790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.767608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:01:55.789144] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927715789137, cur_ts_=1659927715768937, delta=20200) [2022-08-08 11:01:55.820106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:55.840398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:01:55.883008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:56.122335] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=1149] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:01:56.126865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:01:56.135738] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:56.145168] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:56.145194] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:56.163707] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:56.163889] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667601, retry interval=1000000, ret=-4179) [2022-08-08 11:01:56.165824] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:56.165860] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667606, retry interval=1000000, ret=-4179) [2022-08-08 11:01:56.589890] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE3260-0-0] [lt=31] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659927716588409 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927716588399 abs_expired_time=1659927718519063 stmt_timeout=1930664 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21595694 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927716570484 start_version=-1 trans_version=1659927716588399 commit_version=0 stmt_start_time=1659927716580358 abs_expired_time=1659927746481186 stmt_timeout=29900828 abs_lock_wait_timeout=1659927746481186 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:10504424180637063484, inc:84105725, addr:"192.168.11.128:2882", t:1659927716580358} checksum_log_ts=9223372036854775807, version=1659927716588409) [2022-08-08 11:01:56.764758] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] clock out of order(cur_ts=1659927716764753, cur_ts_=1659927716731640, delta=33113) [2022-08-08 11:01:57.151866] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:57.155744] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:57.155783] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:57.206053] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:57.206098] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667605, retry interval=1000000, ret=-4179) [2022-08-08 11:01:57.215444] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:57.215483] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667600, retry interval=1000000, ret=-4179) [2022-08-08 11:01:57.512292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:01:58.158134] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:58.161142] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:58.161165] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:58.247063] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:58.247126] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667599, retry interval=1000000, ret=-4179) [2022-08-08 11:01:58.251999] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:58.252048] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667604, retry interval=1000000, ret=-4179) [2022-08-08 11:01:58.603740] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3266-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:58.603766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3266-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:01:59.163540] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:59.168605] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:01:59.168651] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:01:59.290962] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:59.291282] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] task process failed, start retry(max retry time=9223372036854667598, retry interval=1000000, ret=-4179) [2022-08-08 11:01:59.302627] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:01:59.302661] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667603, retry interval=1000000, ret=-4179) [2022-08-08 11:01:59.531561] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] clock out of order(cur_ts=1659927719531554, cur_ts_=1659927719508778, delta=22776) [2022-08-08 11:01:59.684528] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659927719684523, cur_ts_=1659927719652556, delta=31967) [2022-08-08 11:01:59.690330] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:01:59.718426] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22436) [2022-08-08 11:01:59.794163] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:01:59.794200] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:01:59.908370] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927719908361, cur_ts_=1659927719859437, delta=48924) [2022-08-08 11:01:59.968052] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=25] [dc=0] [trans sampling] (TRACE=begin_ts=1659927719966687 2022-08-08 03:01:59.966687|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:17473868793625188711, inc:84105858, addr:{ip:"192.168.11.128", port:2882}, t:1659927719965000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A65EE-0-0, start_time:1659927719966426, snapshot:1659927719909099|[end_task] u=128 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927719966833|[calc_checksum_by_commit] u=20 ret:0, checksum:4079125820, checksum_log_ts:9223372036854775807|[submit_log] u=51 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1061 trans_version:1659927719966854, uref:1073741825|[exiting] u=29 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8302412, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=1322) [2022-08-08 11:02:00.094887] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927720094882, cur_ts_=1659927720067732, delta=27150) [2022-08-08 11:02:00.170627] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:00.182209] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:00.182279] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10622][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x6d3369c 0x3cd322f 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:00.182291] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14938016}) [2022-08-08 11:02:00.258442] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927720258434, cur_ts_=1659927720230428, delta=28006) [2022-08-08 11:02:00.307428] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A89-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:00.307536] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A89-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:00.348834] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:00.348868] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667597, retry interval=1000000, ret=-4179) [2022-08-08 11:02:00.349670] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:00.349718] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667602, retry interval=1000000, ret=-4179) [2022-08-08 11:02:00.381731] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927718351888, end_time=1659927720381718, elapsed_time=2029830, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:02:00.560991] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1282] [dc=0] clock out of order(cur_ts=1659927720560976, cur_ts_=1659927720537992, delta=22984) [2022-08-08 11:02:00.677683] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927720677678, cur_ts_=1659927720651591, delta=26087) [2022-08-08 11:02:01.038234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.069308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.186247] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:01.189355] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:01.189390] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:01.217314] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.341019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:01.362563] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:01.381173] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:01.381222] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667601, retry interval=1000000, ret=-4179) [2022-08-08 11:02:01.384217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.384222] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:01.384499] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=206] [dc=0] task process failed, start retry(max retry time=9223372036854667596, retry interval=1000000, ret=-4179) [2022-08-08 11:02:01.436835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.482385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.495028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.500869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.503151] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.525111] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:01.605508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.609417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:01.615847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.683833] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE326F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:01.683850] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE326F-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:01.683949] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE326F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:01.684039] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE326F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:01.684269] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE326F-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:01.742735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.766224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.876188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:01.883663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:01.958338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:02.192164] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:02.195219] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:02.195268] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:02.424492] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:02.424740] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=121] [dc=0] task process failed, start retry(max retry time=9223372036854667595, retry interval=1000000, ret=-4179) [2022-08-08 11:02:02.426222] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:02.426271] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667600, retry interval=1000000, ret=-4179) [2022-08-08 11:02:02.769003] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927722768995, cur_ts_=1659927722746629, delta=22366) [2022-08-08 11:02:03.015283] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=2292] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10630, time_dist: generate_cluster_version=10626) [2022-08-08 11:02:03.197554] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:03.202363] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:03.202388] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:03.453764] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:03.453829] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667599, retry interval=1000000, ret=-4179) [2022-08-08 11:02:03.454121] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10331, time_dist: generate_cluster_version=10328) [2022-08-08 11:02:03.454892] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:03.454984] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] task process failed, start retry(max retry time=9223372036854667594, retry interval=1000000, ret=-4179) [2022-08-08 11:02:04.038758] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:02:04.205889] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:04.209440] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:04.209481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:04.234317] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927724234307, cur_ts_=1659927724211175, delta=23132) [2022-08-08 11:02:04.234682] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=20930) [2022-08-08 11:02:04.236071] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=240] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24221, time_dist: generate_cluster_version=24219) [2022-08-08 11:02:04.501136] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:04.501187] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667593, retry interval=1000000, ret=-4179) [2022-08-08 11:02:04.509767] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:04.509833] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667598, retry interval=1000000, ret=-4179) [2022-08-08 11:02:04.720003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:04.812419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:04.822711] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927724822702, cur_ts_=1659927724762924, delta=59778) [2022-08-08 11:02:04.880090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:04.880133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:04.895634] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=81210, time_dist: do_cluster_heartbeat=65, generate_cluster_version=81138) [2022-08-08 11:02:04.977676] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927724977666, cur_ts_=1659927724826101, delta=151565) [2022-08-08 11:02:05.031097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.031142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:05.032790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=237] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.041223] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=145920) [2022-08-08 11:02:05.041262] WARN [CLOG] process_log_items (ob_clog_writer.cpp:262) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] slow flush(flush_time=19, io_time=145945, file_id=237, flush_start_offset=53296154, file_offset=53296776) [2022-08-08 11:02:05.041344] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=24] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:14508911182974417406, inc:84106043, addr:"192.168.11.128:2882", t:1659927724826101}, log_id=8302436, used_time=153149) [2022-08-08 11:02:05.047155] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=165776, time_dist: do_cluster_heartbeat=8, generate_cluster_version=165763) [2022-08-08 11:02:05.056597] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15295) [2022-08-08 11:02:05.057266] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28400, time_dist: do_cluster_heartbeat=9, generate_cluster_version=28389) [2022-08-08 11:02:05.114485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:05.116609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.124699] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=317] [dc=0] clock out of order(cur_ts=1659927725124691, cur_ts_=1659927725083862, delta=40829) [2022-08-08 11:02:05.126906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.130227] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11874) [2022-08-08 11:02:05.130427] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13247, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13234) [2022-08-08 11:02:05.173829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:05.184872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:05.185766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.185865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.186792] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12001) [2022-08-08 11:02:05.187467] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13409, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13396) [2022-08-08 11:02:05.198458] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12449, time_dist: generate_cluster_version=12446) [2022-08-08 11:02:05.233477] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.233876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.234138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:05.236153] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:05.236662] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.239395] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:05.239463] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:05.245513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.245610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.254505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:05.277021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:05.288956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:05.320117] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.326482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.329017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.340882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.340950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.340962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.372122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.372194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.372982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.382465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.383404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.383950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.391829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.392922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.415026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.427030] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.445229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.466529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.488862] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:05.489824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.529446] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927725529441, cur_ts_=1659927725496611, delta=32830) [2022-08-08 11:02:05.564068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.564362] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:05.564671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:05.565224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:05.625265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.631814] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:05.633161] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=467] [dc=0] task process failed, start retry(max retry time=9223372036854667597, retry interval=1000000, ret=-4179) [2022-08-08 11:02:05.636533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:05.645152] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:05.645202] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667592, retry interval=1000000, ret=-4179) [2022-08-08 11:02:05.680063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.689283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.785140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:05.838741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:05.860127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=192] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:05.901750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:06.011765] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927726010762 2022-08-08 03:02:06.010762|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:17576321985248375392, inc:84106082, addr:{ip:"192.168.11.128", port:2882}, t:1659927726010000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC3DC-0-0, start_time:1659927726010652, snapshot:1659927725950986|[end_task] u=208 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927726010987|[calc_checksum_by_commit] u=18 ret:0, checksum:1121019815, checksum_log_ts:9223372036854775807|[submit_log] u=44 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=635 trans_version:1659927726010300, uref:1073741825|[exiting] u=13 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8317037, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=963) [2022-08-08 11:02:06.152235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:06.242960] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:06.247450] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:06.247491] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:06.668091] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11655, time_dist: generate_cluster_version=11651) [2022-08-08 11:02:06.668873] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:06.668922] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667596, retry interval=1000000, ret=-4179) [2022-08-08 11:02:06.673203] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:06.673232] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667591, retry interval=1000000, ret=-4179) [2022-08-08 11:02:06.877789] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927726877769, cur_ts_=1659927726845753, delta=32016) [2022-08-08 11:02:06.907386] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11192, time_dist: do_cluster_heartbeat=13, generate_cluster_version=11175) [2022-08-08 11:02:07.249858] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:07.252936] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:07.252972] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:07.535180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:07.535472] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23543] [dc=0] clock out of order(cur_ts=1659927727535467, cur_ts_=1659927727505304, delta=30163) [2022-08-08 11:02:07.713144] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:07.713197] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667590, retry interval=1000000, ret=-4179) [2022-08-08 11:02:07.717317] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:07.717385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667595, retry interval=1000000, ret=-4179) [2022-08-08 11:02:08.255426] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:08.258696] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:08.258741] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:08.757552] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:08.757608] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667594, retry interval=1000000, ret=-4179) [2022-08-08 11:02:08.761029] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:08.761059] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667589, retry interval=1000000, ret=-4179) [2022-08-08 11:02:08.879280] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37651) [2022-08-08 11:02:08.879505] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38857, time_dist: generate_cluster_version=38854) [2022-08-08 11:02:09.261210] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:09.267223] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:09.267263] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:09.787159] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927729787153, cur_ts_=1659927729760752, delta=26401) [2022-08-08 11:02:09.798989] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:09.819956] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:02:09.819981] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:02:09.828274] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:09.828315] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667588, retry interval=1000000, ret=-4179) [2022-08-08 11:02:09.841722] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:09.841791] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667593, retry interval=1000000, ret=-4179) [2022-08-08 11:02:10.008426] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14942272}) [2022-08-08 11:02:10.197332] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927728352158, end_time=1659927730197321, elapsed_time=1845163, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:02:10.269200] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:10.272758] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:10.272787] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:10.856477] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:10.856511] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667587, retry interval=1000000, ret=-4179) [2022-08-08 11:02:10.865297] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=337] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:10.865343] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667592, retry interval=1000000, ret=-4179) [2022-08-08 11:02:11.054357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.093288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.234773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=194] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.283818] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:11.287019] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:11.287062] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:11.358215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:11.399985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.456543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.503164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.516522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.519934] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.532801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.542222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:11.629572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:11.630535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.636051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.763939] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.784859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.878321] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:11.879614] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] task process failed, start retry(max retry time=9223372036854667586, retry interval=1000000, ret=-4179) [2022-08-08 11:02:11.889516] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:11.889550] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667591, retry interval=1000000, ret=-4179) [2022-08-08 11:02:11.889586] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12060) [2022-08-08 11:02:11.890072] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17326, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17314) [2022-08-08 11:02:11.899752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:11.900421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:11.982293] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:12.049262] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=36] [dc=0] [trans sampling] (TRACE=begin_ts=1659927732048260 2022-08-08 03:02:12.048260|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:11870450933765079135, inc:84106314, addr:{ip:"192.168.11.128", port:2882}, t:1659927732048000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A662A-0-0, start_time:1659927732048145, snapshot:1659927732024092|[end_task] u=121 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927732048400|[calc_checksum_by_commit] u=21 ret:0, checksum:714554515, checksum_log_ts:9223372036854775807|[submit_log] u=66 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=697 trans_version:1659927732048421, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8302472, logging:0, pending:0, uref:1|[end_trans_cb] u=21 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=964) [2022-08-08 11:02:12.291712] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:12.297504] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:12.297541] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:12.688466] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:12.881909] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A9B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:12.882036] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A9B-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:12.882203] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A9B-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:12.882453] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A9B-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:12.882968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4A9B-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:12.914802] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:12.914850] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667590, retry interval=1000000, ret=-4179) [2022-08-08 11:02:12.923687] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:12.923735] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667585, retry interval=1000000, ret=-4179) [2022-08-08 11:02:13.053932] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=339] [dc=0] clock out of order(cur_ts=1659927733053924, cur_ts_=1659927733032680, delta=21244) [2022-08-08 11:02:13.300335] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:13.304493] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:13.305462] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=436] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:13.950190] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:13.950233] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667584, retry interval=1000000, ret=-4179) [2022-08-08 11:02:13.956110] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:13.956170] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667589, retry interval=1000000, ret=-4179) [2022-08-08 11:02:14.308736] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=770] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:14.312212] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:14.312236] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:14.741631] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:14.846914] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:14.898489] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:14.899277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:14.983678] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:14.983732] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667583, retry interval=1000000, ret=-4179) [2022-08-08 11:02:14.989951] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:14.990015] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667588, retry interval=1000000, ret=-4179) [2022-08-08 11:02:15.052224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:15.052678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.053699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.061895] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=7] [dc=0] [trans sampling] (TRACE=begin_ts=1659927735061114 2022-08-08 03:02:15.061114|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:14888340432708985326, inc:84106427, addr:{ip:"192.168.11.128", port:2882}, t:1659927735060000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6639-0-0, start_time:1659927735060958, snapshot:1659927735047507|[end_task] u=148 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927735061280|[calc_checksum_by_commit] u=20 ret:0, checksum:1390551912, checksum_log_ts:9223372036854775807|[submit_log] u=52 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=468 trans_version:1659927735061301, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8302487, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=742) [2022-08-08 11:02:15.132587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=192] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:15.132666] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.147153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.194094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:15.203429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.204412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.209382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:15.254990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.258460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=186] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.258561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:15.258592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.271612] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.271689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.271708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:15.293634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:15.313710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:15.315485] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:15.318631] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:15.318692] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:15.335450] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.350222] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=252] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.351761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.360630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.365142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.365183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.391911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.396466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.396577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.400642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.406903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.406977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.408282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.411941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.435557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.449906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.464421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.497308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.519190] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:15.519416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.544482] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=522] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23766) [2022-08-08 11:02:15.544788] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24853, time_dist: do_cluster_heartbeat=10, generate_cluster_version=24841) [2022-08-08 11:02:15.579653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:15.579849] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.590069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:15.590465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:15.641601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.641638] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927735641634, cur_ts_=1659927735619343, delta=22291) [2022-08-08 11:02:15.652485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=639] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:15.704094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.704901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.809768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:15.860899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.882150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:15.923717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:15.991680] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4ADC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:16.019111] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:16.019145] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667582, retry interval=1000000, ret=-4179) [2022-08-08 11:02:16.026035] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:16.026086] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667587, retry interval=1000000, ret=-4179) [2022-08-08 11:02:16.146750] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:02:16.176846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:16.321462] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:16.324757] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:16.324816] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:17.051922] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:17.051972] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667581, retry interval=1000000, ret=-4179) [2022-08-08 11:02:17.057411] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:17.057474] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667586, retry interval=1000000, ret=-4179) [2022-08-08 11:02:17.328367] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:17.331675] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:17.331732] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:17.551058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:18.078810] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:18.078858] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667580, retry interval=1000000, ret=-4179) [2022-08-08 11:02:18.079781] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=22] [dc=0] [trans sampling] (TRACE=begin_ts=1659927738077652 2022-08-08 03:02:18.077652|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:12983033422012478972, inc:84106536, addr:{ip:"192.168.11.128", port:2882}, t:1659927738072000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC418-0-0, start_time:1659927738077526, snapshot:1659927738074902|[end_task] u=182 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927738077852|[calc_checksum_by_commit] u=20 ret:0, checksum:3988918752, checksum_log_ts:9223372036854775807|[submit_log] u=261 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1558 trans_version:1659927738074903, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8317097, logging:0, pending:0, uref:1|[end_trans_cb] u=32 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=2092) [2022-08-08 11:02:18.093956] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:18.094096] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] task process failed, start retry(max retry time=9223372036854667585, retry interval=1000000, ret=-4179) [2022-08-08 11:02:18.334314] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:18.339436] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:18.339473] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:18.646054] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] clock out of order(cur_ts=1659927738646047, cur_ts_=1659927738615915, delta=30132) [2022-08-08 11:02:19.097177] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:19.097212] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667579, retry interval=1000000, ret=-4179) [2022-08-08 11:02:19.113818] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16518) [2022-08-08 11:02:19.114945] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18344, time_dist: generate_cluster_version=18340) [2022-08-08 11:02:19.121285] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:19.121413] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] task process failed, start retry(max retry time=9223372036854667584, retry interval=1000000, ret=-4179) [2022-08-08 11:02:19.141581] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927739141572, cur_ts_=1659927739121361, delta=20211) [2022-08-08 11:02:19.141628] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10526][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=19059) [2022-08-08 11:02:19.342112] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:19.346253] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:19.346291] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:19.437003] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927739436997, cur_ts_=1659927739413558, delta=23439) [2022-08-08 11:02:19.820759] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:02:19.820780] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:02:19.901574] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:19.936830] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14946656}) [2022-08-08 11:02:19.946340] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13687, time_dist: generate_cluster_version=13683) [2022-08-08 11:02:20.127325] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927738353073, end_time=1659927740127312, elapsed_time=1774239, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:02:20.133342] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:20.133380] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667578, retry interval=1000000, ret=-4179) [2022-08-08 11:02:20.145892] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:20.145941] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667583, retry interval=1000000, ret=-4179) [2022-08-08 11:02:20.348062] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:20.352825] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:20.352849] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:20.864896] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927740864887, cur_ts_=1659927740840029, delta=24858) [2022-08-08 11:02:21.079255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.111860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.164176] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:21.164211] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667577, retry interval=1000000, ret=-4179) [2022-08-08 11:02:21.168314] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:21.168594] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667582, retry interval=1000000, ret=-4179) [2022-08-08 11:02:21.257346] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.356354] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:21.359384] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:21.359426] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:21.382709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:21.417082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=607] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.480403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.520614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.532469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.542254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.548787] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.559341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:21.646697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:21.654034] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.655541] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.782460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.800821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.918213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:21.923918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:21.999411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:22.187968] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:22.188959] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=905] [dc=0] task process failed, start retry(max retry time=9223372036854667576, retry interval=1000000, ret=-4179) [2022-08-08 11:02:22.189782] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:22.189817] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667581, retry interval=1000000, ret=-4179) [2022-08-08 11:02:22.361752] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:22.364856] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:22.364897] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:23.095275] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10116) [2022-08-08 11:02:23.095513] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11033, time_dist: generate_cluster_version=11030) [2022-08-08 11:02:23.217937] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:23.217973] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667575, retry interval=1000000, ret=-4179) [2022-08-08 11:02:23.228155] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:23.228197] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667580, retry interval=1000000, ret=-4179) [2022-08-08 11:02:23.367149] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:23.371806] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:23.371860] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:24.128231] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927744125419 2022-08-08 03:02:24.125419|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:6699315139702771773, inc:84106769, addr:{ip:"192.168.11.128", port:2882}, t:1659927744124000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC436-0-0, start_time:1659927744125314, snapshot:1659927744110688|[end_task] u=175 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927744125611|[calc_checksum_by_commit] u=18 ret:0, checksum:2704925422, checksum_log_ts:9223372036854775807|[submit_log] u=173 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2348 trans_version:1659927744124679, uref:1073741825|[exiting] u=12 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8317127, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=2773) [2022-08-08 11:02:24.287358] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927744287353, cur_ts_=1659927744247555, delta=39798) [2022-08-08 11:02:24.297154] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:24.297187] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667579, retry interval=1000000, ret=-4179) [2022-08-08 11:02:24.299523] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:24.299571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667574, retry interval=1000000, ret=-4179) [2022-08-08 11:02:24.374576] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:24.377784] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:24.377819] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:24.758543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:24.862634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:24.916036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:24.919003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.071451] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.076536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.076671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:25.163058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:25.163671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.163789] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.163874] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927745163872, cur_ts_=1659927745137862, delta=26010) [2022-08-08 11:02:25.216282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:25.225288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.225430] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.225473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:25.277410] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.277947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:25.278028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.278227] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.288075] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.288306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.288349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:25.310009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:25.325594] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:25.325660] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667578, retry interval=1000000, ret=-4179) [2022-08-08 11:02:25.331989] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:25.332029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667573, retry interval=1000000, ret=-4179) [2022-08-08 11:02:25.335720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:25.356616] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.368551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.371306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.377553] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.380163] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:25.383666] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:25.383712] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:25.387963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.388150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.413402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.414333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.419408] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=1054] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.420657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.423590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.423683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.430563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.432051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.454578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=189] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.473689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.482170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.517089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.544178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:25.544623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.596013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.604847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:25.608849] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:25.610826] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:25.658466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.668627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=403] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:25.722906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.734170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.830565] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:25.877354] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.901552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:25.941207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:25.981915] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659927745981908, cur_ts_=1659927745954177, delta=27731) [2022-08-08 11:02:26.124964] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16574, time_dist: generate_cluster_version=16569) [2022-08-08 11:02:26.195187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:26.347872] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:26.347908] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667577, retry interval=1000000, ret=-4179) [2022-08-08 11:02:26.359041] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:26.359146] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] task process failed, start retry(max retry time=9223372036854667572, retry interval=1000000, ret=-4179) [2022-08-08 11:02:26.389329] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:26.393556] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:26.393605] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:27.361088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AB0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:27.361766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AB0-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:27.362360] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AB0-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:27.362655] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AB0-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:27.362738] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AB0-0-0] [lt=67] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:27.382308] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:27.382360] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667576, retry interval=1000000, ret=-4179) [2022-08-08 11:02:27.384518] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:27.384566] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667571, retry interval=1000000, ret=-4179) [2022-08-08 11:02:27.408172] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:27.411215] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:27.411252] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:27.574627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:27.888113] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32BD-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.888144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32BD-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.888344] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32BD-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.888551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32BD-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.888795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.888968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.889084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:27.889204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:28.413954] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:28.420508] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:28.420532] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:28.440572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:28.440591] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:28.441888] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:28.441926] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667570, retry interval=1000000, ret=-4179) [2022-08-08 11:02:28.443373] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:28.443405] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667575, retry interval=1000000, ret=-4179) [2022-08-08 11:02:28.813845] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12121) [2022-08-08 11:02:28.833054] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18090) [2022-08-08 11:02:28.953031] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12410) [2022-08-08 11:02:28.953260] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14086, time_dist: generate_cluster_version=14083) [2022-08-08 11:02:28.979794] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10568) [2022-08-08 11:02:28.980082] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12098, time_dist: generate_cluster_version=12095) [2022-08-08 11:02:29.193971] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927749193965, cur_ts_=1659927749159465, delta=34500) [2022-08-08 11:02:29.207009] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=37667, time_dist: generate_cluster_version=37663) [2022-08-08 11:02:29.395023] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=416] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20098) [2022-08-08 11:02:29.395232] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21092, time_dist: generate_cluster_version=21090) [2022-08-08 11:02:29.424185] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:29.427425] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:29.427451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:29.487507] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:29.487540] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667574, retry interval=1000000, ret=-4179) [2022-08-08 11:02:29.489274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:29.490947] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:29.490984] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667569, retry interval=1000000, ret=-4179) [2022-08-08 11:02:29.559797] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15016) [2022-08-08 11:02:29.560087] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16020, time_dist: do_cluster_heartbeat=11, generate_cluster_version=16007) [2022-08-08 11:02:29.696425] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659927749696420, cur_ts_=1659927749670898, delta=25522) [2022-08-08 11:02:29.811464] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:02:29.811501] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:02:29.938105] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] clock out of order(cur_ts=1659927749938098, cur_ts_=1659927749911330, delta=26768) [2022-08-08 11:02:29.944910] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32928) [2022-08-08 11:02:29.945349] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=34282, time_dist: do_cluster_heartbeat=10, generate_cluster_version=34269) [2022-08-08 11:02:29.965954] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=187] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17644) [2022-08-08 11:02:29.966738] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19727, time_dist: generate_cluster_version=19723) [2022-08-08 11:02:29.997077] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:30.001212] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14951008}) [2022-08-08 11:02:30.167900] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659927750164774 2022-08-08 03:02:30.164774|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:4953991471657719350, inc:84107004, addr:{ip:"192.168.11.128", port:2882}, t:1659927750164000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6685-0-0, start_time:1659927750164625, snapshot:1659927750070566|[end_task] u=128 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927750164918|[calc_checksum_by_commit] u=21 ret:0, checksum:4091310684, checksum_log_ts:9223372036854775807|[submit_log] u=51 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2814 trans_version:1659927750164940, uref:1073741825|[exiting] u=26 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8302563, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=3083) [2022-08-08 11:02:30.207589] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927748353721, end_time=1659927750207575, elapsed_time=1853854, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:02:30.429299] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:30.434260] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:30.434304] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:30.501401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.501434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.501498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.501656] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.501907] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.502175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.503372] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62BA-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:30.525638] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:30.525721] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667568, retry interval=1000000, ret=-4179) [2022-08-08 11:02:30.526559] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:30.526587] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667573, retry interval=1000000, ret=-4179) [2022-08-08 11:02:30.913277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32C6-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.913334] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32C6-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.913459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32C6-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.913590] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32C6-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.913730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.913930] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.914099] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.914184] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:30.914218] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:31.098756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.135919] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.231287] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659927751231275, cur_ts_=1659927751207812, delta=23463) [2022-08-08 11:02:31.281095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.405524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:31.432063] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13506, time_dist: generate_cluster_version=13500) [2022-08-08 11:02:31.437016] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:31.443505] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:31.443549] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:31.446631] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:31.504477] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:31.540217] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.550331] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.560149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.561690] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:31.561754] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667567, retry interval=1000000, ret=-4179) [2022-08-08 11:02:31.566237] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:31.566431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:31.567535] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:31.567604] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667572, retry interval=1000000, ret=-4179) [2022-08-08 11:02:31.570664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:31.578333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:31.664147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:31.669182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:31.674726] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:31.804156] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.825981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:31.939469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:31.943055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:32.017540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:32.446970] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:32.453110] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:32.453178] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:32.597023] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:32.597103] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667571, retry interval=1000000, ret=-4179) [2022-08-08 11:02:32.597794] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:32.598013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:32.599291] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:32.599343] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667566, retry interval=1000000, ret=-4179) [2022-08-08 11:02:32.688882] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:33.455717] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:33.459192] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:33.459386] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:33.627250] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:33.627300] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667570, retry interval=1000000, ret=-4179) [2022-08-08 11:02:33.628351] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:33.630584] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:33.630615] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667565, retry interval=1000000, ret=-4179) [2022-08-08 11:02:33.682715] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927753682707, cur_ts_=1659927753655217, delta=27490) [2022-08-08 11:02:34.465183] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:34.468458] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:34.468495] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:34.631075] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.631102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.635172] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62C0-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.635197] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62C0-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.635205] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62C0-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.661579] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:34.661612] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667569, retry interval=1000000, ret=-4179) [2022-08-08 11:02:34.663152] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.663188] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:34.664304] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:34.664337] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667564, retry interval=1000000, ret=-4179) [2022-08-08 11:02:34.777181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:34.880433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:34.932624] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=201] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:34.944361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.089800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.100279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.100411] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:35.183550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.183685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.186938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=558] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:35.246302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:35.248999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:35.249157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.256602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.299526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.299658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:35.306675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.306696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.307656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=2115] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:35.307751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.310854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.334803] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=169] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:35.361099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:35.380279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.386825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.394961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=125] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.395038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.404883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.413042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.432055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.436032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.436338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.438836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.444347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.446388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.446438] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.454688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.471053] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:35.474202] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:35.474237] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:35.475077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.489060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.499009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.533064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.561535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.562484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:35.617002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.620206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=236] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:35.626833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:35.628371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:35.677583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.685436] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:35.685470] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667568, retry interval=1000000, ret=-4179) [2022-08-08 11:02:35.692153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:35.693590] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:35.693623] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667563, retry interval=1000000, ret=-4179) [2022-08-08 11:02:35.738597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.750343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.847152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:35.902751] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:35.925620] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:35.956589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:36.173718] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:02:36.175433] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13761) [2022-08-08 11:02:36.175656] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14762, time_dist: generate_cluster_version=14759) [2022-08-08 11:02:36.178407] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13355, time_dist: generate_cluster_version=13352) [2022-08-08 11:02:36.212836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:36.478459] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:36.481947] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:36.482037] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:36.605981] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927756605976, cur_ts_=1659927756584805, delta=21171) [2022-08-08 11:02:36.714577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:36.714619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:36.716176] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:36.716227] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667567, retry interval=1000000, ret=-4179) [2022-08-08 11:02:36.716900] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:36.716945] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667562, retry interval=1000000, ret=-4179) [2022-08-08 11:02:37.484869] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:37.488012] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:37.488064] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:37.597989] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:37.742924] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:37.742984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:37.747017] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:37.747060] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667566, retry interval=1000000, ret=-4179) [2022-08-08 11:02:37.756751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:37.756797] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:37.758553] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:37.758652] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667561, retry interval=1000000, ret=-4179) [2022-08-08 11:02:38.490775] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:38.501909] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:38.501935] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:38.792913] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] clock out of order(cur_ts=1659927758792905, cur_ts_=1659927758757820, delta=35085) [2022-08-08 11:02:38.798506] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15734, time_dist: do_cluster_heartbeat=9, generate_cluster_version=15723) [2022-08-08 11:02:38.812287] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:38.812324] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667565, retry interval=1000000, ret=-4179) [2022-08-08 11:02:38.813176] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:38.813207] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:38.814806] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:38.814842] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667560, retry interval=1000000, ret=-4179) [2022-08-08 11:02:39.506598] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:39.509560] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:39.509584] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:39.595118] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927759595109, cur_ts_=1659927759570520, delta=24589) [2022-08-08 11:02:39.596178] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=22103) [2022-08-08 11:02:39.619128] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22990) [2022-08-08 11:02:39.621533] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=140] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=48326, time_dist: generate_cluster_version=48323) [2022-08-08 11:02:39.628446] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=33272, time_dist: generate_cluster_version=33267) [2022-08-08 11:02:39.812940] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:02:39.812956] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:02:39.854970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:39.855008] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:39.856012] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:39.856075] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667559, retry interval=1000000, ret=-4179) [2022-08-08 11:02:39.857721] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:39.857758] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667564, retry interval=1000000, ret=-4179) [2022-08-08 11:02:39.921956] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927759921952, cur_ts_=1659927759901875, delta=20077) [2022-08-08 11:02:39.979967] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15879) [2022-08-08 11:02:39.980337] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17055, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17043) [2022-08-08 11:02:39.981377] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17273, time_dist: do_cluster_heartbeat=7, generate_cluster_version=17265) [2022-08-08 11:02:39.981740] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32E1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:39.981972] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE32E1-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:39.982420] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:39.982587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:39.982852] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:39.982868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:40.021766] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14955392}) [2022-08-08 11:02:40.076327] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:40.211237] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927758354311, end_time=1659927760211224, elapsed_time=1856913, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:02:40.510951] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:40.516030] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:40.516065] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:40.552680] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927760552672, cur_ts_=1659927760524078, delta=28594) [2022-08-08 11:02:40.885030] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:40.885384] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=306] [dc=0] task process failed, start retry(max retry time=9223372036854667563, retry interval=1000000, ret=-4179) [2022-08-08 11:02:40.885310] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:40.885615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=290] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:40.887134] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:40.887180] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667558, retry interval=1000000, ret=-4179) [2022-08-08 11:02:41.113662] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927761113653, cur_ts_=1659927761086584, delta=27069) [2022-08-08 11:02:41.123628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.155773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.303643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.428695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:41.472877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:41.519685] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:41.521984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:41.532540] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:41.532582] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:41.561777] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.573462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.583654] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.593870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:41.593966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:41.684802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:41.685310] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:41.704912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:41.794844] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927761794833, cur_ts_=1659927761760484, delta=34349) [2022-08-08 11:02:41.828375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.866178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:41.897264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:41.897288] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:41.939125] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:41.939193] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667562, retry interval=1000000, ret=-4179) [2022-08-08 11:02:41.947181] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:41.947297] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667557, retry interval=1000000, ret=-4179) [2022-08-08 11:02:41.955839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:41.966928] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:42.039888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:42.083180] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927762083175, cur_ts_=1659927762056756, delta=26419) [2022-08-08 11:02:42.534704] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:42.538333] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:42.538370] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:42.949143] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B03-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:42.949165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B03-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:42.969392] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:42.969432] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667561, retry interval=1000000, ret=-4179) [2022-08-08 11:02:42.970779] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:42.971016] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:42.972464] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:42.972522] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667556, retry interval=1000000, ret=-4179) [2022-08-08 11:02:43.313762] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927763313753, cur_ts_=1659927763293587, delta=20166) [2022-08-08 11:02:43.541653] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:43.545797] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:43.545843] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:43.876881] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927763876872, cur_ts_=1659927763855669, delta=21203) [2022-08-08 11:02:44.041081] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927764041070, cur_ts_=1659927764009774, delta=31296) [2022-08-08 11:02:44.047923] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:44.047964] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667555, retry interval=1000000, ret=-4179) [2022-08-08 11:02:44.056830] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=145] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10639) [2022-08-08 11:02:44.057088] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14898, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14886) [2022-08-08 11:02:44.057742] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:44.057778] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667560, retry interval=1000000, ret=-4179) [2022-08-08 11:02:44.076059] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23718, time_dist: generate_cluster_version=23716) [2022-08-08 11:02:44.099917] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23720) [2022-08-08 11:02:44.247986] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14380) [2022-08-08 11:02:44.248201] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15323, time_dist: generate_cluster_version=15318, cluster_service_self_check=3) [2022-08-08 11:02:44.250864] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17812, time_dist: generate_cluster_version=17808) [2022-08-08 11:02:44.548671] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:44.551807] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:44.551845] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:44.799485] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:44.903570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=220] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:44.952785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:44.965561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.052840] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62CF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.052898] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62CF-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.052920] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62CF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.053221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62CF-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.057721] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.057753] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.060862] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B06-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.060921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B06-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.060947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B06-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.060964] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B06-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.060976] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B06-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.060999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B06-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.094111] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.095581] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:45.095639] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667559, retry interval=1000000, ret=-4179) [2022-08-08 11:02:45.099650] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.099816] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:45.107004] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:45.107062] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667554, retry interval=1000000, ret=-4179) [2022-08-08 11:02:45.119846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.119865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:45.121837] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.205165] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.205516] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.205603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:45.266164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:45.267188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.268055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:45.278753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.318600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.320468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:45.328486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.329534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.329572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.330603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:45.331580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.352801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:45.383875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:45.401405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.411883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.411942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.412061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.422387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.432763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.451644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.454555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=118] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.456518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.458469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.464624] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.464950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.468929] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.474906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.496167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.513567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.519960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.555921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.557270] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:45.561932] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:45.562047] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:45.580057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.586508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:45.632562] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=147] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.640474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:45.644991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:45.650974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:45.703083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.707558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:45.760202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.771941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.865753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:45.919521] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=318] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:45.949559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:45.980792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:46.090138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:46.090340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:46.090433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:46.110594] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4ACB-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:46.131057] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:46.131100] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667558, retry interval=1000000, ret=-4179) [2022-08-08 11:02:46.136091] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:46.136132] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667553, retry interval=1000000, ret=-4179) [2022-08-08 11:02:46.234974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:46.565059] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:46.568177] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:46.568214] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:47.224514] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:47.224560] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667552, retry interval=1000000, ret=-4179) [2022-08-08 11:02:47.230665] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:47.230826] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:47.232256] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:47.232291] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667557, retry interval=1000000, ret=-4179) [2022-08-08 11:02:47.570696] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:47.574863] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:47.574890] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:47.613319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:48.098480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:48.235120] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:48.235145] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:48.235150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:48.259646] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:48.259735] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667551, retry interval=1000000, ret=-4179) [2022-08-08 11:02:48.270883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:48.271028] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:48.272743] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:48.272787] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667556, retry interval=1000000, ret=-4179) [2022-08-08 11:02:48.577318] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:48.582906] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:48.583013] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:48.850730] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10513, time_dist: generate_cluster_version=10508) [2022-08-08 11:02:48.939031] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927768939021, cur_ts_=1659927768918793, delta=20228) [2022-08-08 11:02:49.268228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62D7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:49.268280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62D7-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:49.268306] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62D7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:49.268319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62D7-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:49.268631] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62D7-0-0] [lt=292] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:49.295999] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:49.296113] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667550, retry interval=1000000, ret=-4179) [2022-08-08 11:02:49.304481] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:49.304512] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667555, retry interval=1000000, ret=-4179) [2022-08-08 11:02:49.585435] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:49.589807] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:49.589833] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:49.660212] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12088) [2022-08-08 11:02:49.660481] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15016, time_dist: generate_cluster_version=15015) [2022-08-08 11:02:49.815110] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:02:49.815469] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=354] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:02:50.009805] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14959744}) [2022-08-08 11:02:50.149565] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:50.198413] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927768355365, end_time=1659927770198401, elapsed_time=1843036, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:02:50.329617] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:50.329666] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667549, retry interval=1000000, ret=-4179) [2022-08-08 11:02:50.345110] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:50.345179] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667554, retry interval=1000000, ret=-4179) [2022-08-08 11:02:50.591753] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:50.595191] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:50.595216] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:50.867401] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927770867391, cur_ts_=1659927770847294, delta=20097) [2022-08-08 11:02:51.142456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.173699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.328897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.377138] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2569] [dc=0] clock out of order(cur_ts=1659927771377132, cur_ts_=1659927771345804, delta=31328) [2022-08-08 11:02:51.402353] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:51.402386] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667548, retry interval=1000000, ret=-4179) [2022-08-08 11:02:51.406981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:51.407005] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:51.408402] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:51.408433] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667553, retry interval=1000000, ret=-4179) [2022-08-08 11:02:51.455877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:51.456638] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20496) [2022-08-08 11:02:51.457320] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22198, time_dist: do_cluster_heartbeat=12, generate_cluster_version=22183) [2022-08-08 11:02:51.497242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:51.500743] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927771500734, cur_ts_=1659927771466791, delta=33943) [2022-08-08 11:02:51.539349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:51.581538] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.590940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.597772] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:51.601091] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:51.601182] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:51.604254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.614597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:51.614706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:51.708459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:51.710365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:51.720288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:51.847287] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.886776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:51.980407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:51.990556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:52.063546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:52.317267] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927772317257, cur_ts_=1659927772290452, delta=26805) [2022-08-08 11:02:52.410285] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:52.410438] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD4-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:52.445864] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:52.445886] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:52.448992] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:52.449033] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667547, retry interval=1000000, ret=-4179) [2022-08-08 11:02:52.449844] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10167, time_dist: generate_cluster_version=10164) [2022-08-08 11:02:52.451183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:52.452051] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:52.458895] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:52.458944] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667552, retry interval=1000000, ret=-4179) [2022-08-08 11:02:52.605590] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:52.609289] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:52.609325] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:52.689471] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:52.919336] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18413) [2022-08-08 11:02:52.919668] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19550, time_dist: generate_cluster_version=19547) [2022-08-08 11:02:53.158615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:53.158781] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:53.475877] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:53.476954] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] task process failed, start retry(max retry time=9223372036854667546, retry interval=1000000, ret=-4179) [2022-08-08 11:02:53.486268] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:53.486318] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667551, retry interval=1000000, ret=-4179) [2022-08-08 11:02:53.611494] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:53.614752] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:53.614802] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:54.489597] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.489935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD7-0-0] [lt=123] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.489955] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD7-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.489968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.490012] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD7-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.490023] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4AD7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.508470] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.508665] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:54.509610] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:54.509670] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667545, retry interval=1000000, ret=-4179) [2022-08-08 11:02:54.513052] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:54.513103] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667550, retry interval=1000000, ret=-4179) [2022-08-08 11:02:54.617385] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:54.623247] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:54.623289] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:54.816283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:54.921469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:54.973536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:54.984019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.140385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=906] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.140440] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=927] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:55.146159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=235] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.228476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.229996] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.230945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:55.283522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:55.283599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.285055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:55.301866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.340592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.341532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:55.345989] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.349971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.351208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:55.353950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.354121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.371003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:02:55.406081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:55.423879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.431815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.432867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.432913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.444974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.458210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.471100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.477014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.478723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.479441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.486438] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=190] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.487836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.491866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.493335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.512454] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E0-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.512585] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E0-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.512776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E0-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.512905] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E0-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.513170] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E0-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.513229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E0-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.522691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.532125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.539849] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:55.540066] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667544, retry interval=1000000, ret=-4179) [2022-08-08 11:02:55.543942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.544494] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.544709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:55.545860] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:55.545908] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667549, retry interval=1000000, ret=-4179) [2022-08-08 11:02:55.579888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.600618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.610994] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=202] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:55.625743] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:55.629146] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:55.629303] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:55.653608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.663476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:02:55.663552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:55.673966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:02:55.725623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.727933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:55.779054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.795900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.888721] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:02:55.937891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:55.969103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:02:55.996464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:56.197275] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:02:56.257520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:02:56.568794] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:56.568830] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667548, retry interval=1000000, ret=-4179) [2022-08-08 11:02:56.570730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:56.571000] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:02:56.572331] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:56.572365] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667543, retry interval=1000000, ret=-4179) [2022-08-08 11:02:56.631806] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:56.635140] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:56.635176] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:57.228997] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] clock out of order(cur_ts=1659927777228990, cur_ts_=1659927777207172, delta=21818) [2022-08-08 11:02:57.633770] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:02:57.637373] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:57.637458] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667542, retry interval=1000000, ret=-4179) [2022-08-08 11:02:57.639716] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:57.653516] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:57.653568] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667547, retry interval=1000000, ret=-4179) [2022-08-08 11:02:57.653773] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10357) [2022-08-08 11:02:57.657328] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:57.657371] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:58.038818] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927778038792, cur_ts_=1659927778006075, delta=32717) [2022-08-08 11:02:58.622336] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] clock out of order(cur_ts=1659927778622326, cur_ts_=1659927778599042, delta=23284) [2022-08-08 11:02:58.663014] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:58.674891] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:58.674915] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:58.686043] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:58.686114] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667541, retry interval=1000000, ret=-4179) [2022-08-08 11:02:58.697392] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:58.697515] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667546, retry interval=1000000, ret=-4179) [2022-08-08 11:02:59.195639] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331A-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:59.195900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:02:59.397130] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927779397125, cur_ts_=1659927779375615, delta=21510) [2022-08-08 11:02:59.677294] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:02:59.683497] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:02:59.683549] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:02:59.723723] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:59.723776] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667540, retry interval=1000000, ret=-4179) [2022-08-08 11:02:59.725315] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=871] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:02:59.725439] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667545, retry interval=1000000, ret=-4179) [2022-08-08 11:02:59.725722] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14364) [2022-08-08 11:02:59.726308] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24444, time_dist: do_cluster_heartbeat=6, generate_cluster_version=24436) [2022-08-08 11:02:59.812446] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=453] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:02:59.812483] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:00.033316] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927780033306, cur_ts_=1659927780010286, delta=23020) [2022-08-08 11:03:00.087264] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16255) [2022-08-08 11:03:00.090733] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20470, time_dist: generate_cluster_version=20466) [2022-08-08 11:03:00.129866] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19702) [2022-08-08 11:03:00.130368] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21991, time_dist: generate_cluster_version=21988) [2022-08-08 11:03:00.160107] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14964064}) [2022-08-08 11:03:00.204318] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.204340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.204346] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.204350] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.204367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.204376] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.204380] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE331D-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:00.215054] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:00.358214] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927778355955, end_time=1659927780358200, elapsed_time=2002245, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:03:00.685205] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:00.688533] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:00.688712] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:00.755000] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11632, time_dist: generate_cluster_version=11630) [2022-08-08 11:03:00.755880] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:00.755915] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667539, retry interval=1000000, ret=-4179) [2022-08-08 11:03:00.761943] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:00.763769] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:00.763823] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667544, retry interval=1000000, ret=-4179) [2022-08-08 11:03:01.164376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.194115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.344393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.473290] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=510] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:01.516586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:01.560827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:01.601742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.606285] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.623070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.632015] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:01.632226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:01.691047] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:01.694237] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:01.694263] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:01.726484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:01.726660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:01.738289] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:01.760074] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.765662] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.765929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.766019] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.766150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.766242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.766459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.766740] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.766777] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62E9-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:01.789509] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:01.789545] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667543, retry interval=1000000, ret=-4179) [2022-08-08 11:03:01.794290] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:01.794331] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667538, retry interval=1000000, ret=-4179) [2022-08-08 11:03:01.863159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:01.902459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:01.999729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:02.006282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:02.080118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:02.136204] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927782135269, cur_ts_=1659927782101094, delta=34175) [2022-08-08 11:03:02.720645] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:02.722324] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] clock out of order(cur_ts=1659927782722318, cur_ts_=1659927782688050, delta=34268) [2022-08-08 11:03:02.728321] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:02.728359] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:02.850432] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:02.850475] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667537, retry interval=1000000, ret=-4179) [2022-08-08 11:03:02.856943] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:02.857175] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=150] [dc=0] task process failed, start retry(max retry time=9223372036854667542, retry interval=1000000, ret=-4179) [2022-08-08 11:03:02.922272] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] clock out of order(cur_ts=1659927782922267, cur_ts_=1659927782888567, delta=33700) [2022-08-08 11:03:03.076294] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] clock out of order(cur_ts=1659927783076285, cur_ts_=1659927783037556, delta=38729) [2022-08-08 11:03:03.731713] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:03.734976] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:03.735079] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:03.853922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62EC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:03.853992] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62EC-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:03.854024] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62EC-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:03.854244] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62EC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:03.854531] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62EC-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:03.879009] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:03.879058] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667541, retry interval=1000000, ret=-4179) [2022-08-08 11:03:03.882306] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:03.882390] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667536, retry interval=1000000, ret=-4179) [2022-08-08 11:03:04.010570] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927784010541, cur_ts_=1659927783981656, delta=28885) [2022-08-08 11:03:04.136878] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:03:04.242652] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3329-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:04.242730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3329-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:04.243025] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:04.243169] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:04.243282] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:04.737511] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:04.740878] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:04.740923] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:04.839698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:04.908307] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:04.908528] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] task process failed, start retry(max retry time=9223372036854667540, retry interval=1000000, ret=-4179) [2022-08-08 11:03:04.913617] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:04.913731] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:04.918212] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:04.918276] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667535, retry interval=1000000, ret=-4179) [2022-08-08 11:03:04.942906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=281] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:04.996784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.004870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=162] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.162612] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.164592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.164730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:05.252866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.252959] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.253049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:05.304152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.304386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:05.304712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:05.328806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.359571] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.361511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:05.370600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.372302] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.372378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.372679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:05.378943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.392368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:05.418264] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927785418254, cur_ts_=1659927785385175, delta=33079) [2022-08-08 11:03:05.426598] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:05.446460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.451220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.451467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.457960] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.469931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.480978] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.492514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.499397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.499559] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.502670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.505376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.505798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.508974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.513323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.549152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:05.557069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.561349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.601911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.616963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.642464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:05.642583] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659927785642578, cur_ts_=1659927785618434, delta=24144) [2022-08-08 11:03:05.673742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.684136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:05.686009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:05.694535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:05.743531] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:05.746311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:05.748674] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:05.748729] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:05.748809] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.756967] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10642, time_dist: do_cluster_heartbeat=11, generate_cluster_version=10627) [2022-08-08 11:03:05.757943] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10670, time_dist: generate_cluster_version=10667) [2022-08-08 11:03:05.802899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.820807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.904899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:05.934901] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:05.934949] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667539, retry interval=1000000, ret=-4179) [2022-08-08 11:03:05.943682] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:05.943848] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667534, retry interval=1000000, ret=-4179) [2022-08-08 11:03:05.962225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:05.993508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:06.011793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:06.256993] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE332F-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:06.257041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE332F-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:06.257058] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE332F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:06.257067] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE332F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:06.257156] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:06.257223] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:06.274714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:06.751156] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:06.754396] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:06.754433] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:06.963447] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:06.963561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:06.965272] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:06.965308] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667538, retry interval=1000000, ret=-4179) [2022-08-08 11:03:06.967718] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:06.967749] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667533, retry interval=1000000, ret=-4179) [2022-08-08 11:03:07.655745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:07.756558] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:07.760040] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:07.760149] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:07.998582] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13461, time_dist: generate_cluster_version=13457) [2022-08-08 11:03:08.029424] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=818] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:08.030426] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=594] [dc=0] task process failed, start retry(max retry time=9223372036854667537, retry interval=1000000, ret=-4179) [2022-08-08 11:03:08.031246] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32948, time_dist: generate_cluster_version=32944) [2022-08-08 11:03:08.038170] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:08.038904] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:08.042807] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:08.042876] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667532, retry interval=1000000, ret=-4179) [2022-08-08 11:03:08.281869] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3335-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.281921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3335-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.281940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3335-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.281949] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3335-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.282046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.282062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.282282] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.282590] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=258] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:08.466519] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927788466513, cur_ts_=1659927788442397, delta=24116) [2022-08-08 11:03:08.584821] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12605) [2022-08-08 11:03:08.585975] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14529, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14516) [2022-08-08 11:03:08.604940] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=147] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19939) [2022-08-08 11:03:08.605159] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32944, time_dist: do_cluster_heartbeat=6, generate_cluster_version=32935) [2022-08-08 11:03:08.765089] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:08.769279] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:08.769316] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:09.014612] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927789014606, cur_ts_=1659927788989222, delta=25384) [2022-08-08 11:03:09.071352] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:09.071385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667531, retry interval=1000000, ret=-4179) [2022-08-08 11:03:09.079680] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:09.079715] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667536, retry interval=1000000, ret=-4179) [2022-08-08 11:03:09.194921] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659927789194916, cur_ts_=1659927789174130, delta=20786) [2022-08-08 11:03:09.360089] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10962, time_dist: do_cluster_heartbeat=9, generate_cluster_version=10948) [2022-08-08 11:03:09.412396] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=14890) [2022-08-08 11:03:09.413471] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16670, time_dist: do_cluster_heartbeat=13, generate_cluster_version=16654) [2022-08-08 11:03:09.793947] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:09.798145] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:09.798193] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:09.821579] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:03:09.821617] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:10.037077] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14968416}) [2022-08-08 11:03:10.087893] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088233] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=308] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088241] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088251] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088269] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.088280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089421] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089427] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089445] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089464] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089474] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089480] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC62F5-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.089638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.092293] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.092339] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:10.145272] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927790145266, cur_ts_=1659927790117247, delta=28019) [2022-08-08 11:03:10.159478] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:10.159512] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667535, retry interval=1000000, ret=-4179) [2022-08-08 11:03:10.166192] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:10.166275] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667530, retry interval=1000000, ret=-4179) [2022-08-08 11:03:10.186227] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20273) [2022-08-08 11:03:10.281355] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:10.282917] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927788356083, end_time=1659927790282908, elapsed_time=1926825, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:03:10.301419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:10.799660] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:10.802784] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:10.802819] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:11.182828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.188027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:11.189544] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:11.189648] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667534, retry interval=1000000, ret=-4179) [2022-08-08 11:03:11.207391] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:11.207430] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667529, retry interval=1000000, ret=-4179) [2022-08-08 11:03:11.218835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.362922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.494065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:11.534443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:11.580470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:11.622150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.625268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.648592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.649023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:11.655301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:11.750066] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:11.750155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=255] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:11.760481] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:11.808020] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:11.811562] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:11.811604] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:11.883479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:11.919536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:12.016317] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:12.021887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:12.095748] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:12.209580] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:12.209672] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:12.213606] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:12.213650] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667533, retry interval=1000000, ret=-4179) [2022-08-08 11:03:12.232885] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:12.232937] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667528, retry interval=1000000, ret=-4179) [2022-08-08 11:03:12.323553] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3341-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:12.333031] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:12.689986] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:12.814422] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:12.817758] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:12.817827] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:13.255413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:13.255557] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:13.279238] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:13.279275] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667532, retry interval=1000000, ret=-4179) [2022-08-08 11:03:13.289559] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:13.290018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:13.294420] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:13.294709] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=232] [dc=0] task process failed, start retry(max retry time=9223372036854667527, retry interval=1000000, ret=-4179) [2022-08-08 11:03:13.820326] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:13.824844] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:13.825365] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=505] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:14.296760] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B30-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:14.296802] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B30-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:14.302321] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:14.302368] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667531, retry interval=1000000, ret=-4179) [2022-08-08 11:03:14.325918] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:14.325984] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667526, retry interval=1000000, ret=-4179) [2022-08-08 11:03:14.346629] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3347-0-0] [lt=642] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.346655] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3347-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.346969] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3347-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.347111] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3347-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.347210] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.347412] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.347684] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.347710] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:14.829748] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=198] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:14.834779] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:14.834803] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:14.858510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:14.961041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.016071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.026213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.185383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.185245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.187012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:15.269215] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.269275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.273124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:15.309661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:15.309697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:15.319318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.322208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:15.333190] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:15.340417] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:15.340671] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667530, retry interval=1000000, ret=-4179) [2022-08-08 11:03:15.348989] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.357988] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:15.358065] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:15.359919] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:15.359980] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667525, retry interval=1000000, ret=-4179) [2022-08-08 11:03:15.384318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.386746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:15.387685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.393168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.394547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.394619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:15.402552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.408555] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:15.441784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:15.462676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.477984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=135] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.478026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.481112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.491510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.501885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.514813] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.519781] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.519954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.520158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.522739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.530285] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.530354] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.530391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.570976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:15.572853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.583285] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.623421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.633722] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.664471] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:15.689127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.699316] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:15.709871] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:15.710121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:15.762020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:15.772248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.825547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.839443] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:15.842899] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:15.843021] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:15.845520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:15.922408] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=204] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:15.985954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:16.015264] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:16.036069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:16.115211] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927796115202, cur_ts_=1659927796079632, delta=35570) [2022-08-08 11:03:16.223001] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:03:16.292124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:16.364330] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:16.364384] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667529, retry interval=1000000, ret=-4179) [2022-08-08 11:03:16.387921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:16.388227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=288] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:16.389423] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:16.389471] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667524, retry interval=1000000, ret=-4179) [2022-08-08 11:03:16.845420] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:16.849484] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:16.849522] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:17.389245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:17.389266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:17.393040] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:17.393086] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667528, retry interval=1000000, ret=-4179) [2022-08-08 11:03:17.411477] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:17.411596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:17.413462] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=182] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:17.413544] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667523, retry interval=1000000, ret=-4179) [2022-08-08 11:03:17.671956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:17.855289] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:17.858380] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:17.858450] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:18.412094] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18175) [2022-08-08 11:03:18.412382] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19276, time_dist: print_stat=7, generate_cluster_version=19264, cluster_service_self_check=4) [2022-08-08 11:03:18.473010] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927798473003, cur_ts_=1659927798445240, delta=27763) [2022-08-08 11:03:18.505756] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:18.505791] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667527, retry interval=1000000, ret=-4179) [2022-08-08 11:03:18.538148] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:18.538182] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667522, retry interval=1000000, ret=-4179) [2022-08-08 11:03:18.539513] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32685) [2022-08-08 11:03:18.539647] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927798506307 2022-08-08 03:03:18.506307|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:8386257765904521497, inc:84108823, addr:{ip:"192.168.11.128", port:2882}, t:1659927798505000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7F70-0-0, start_time:1659927798506194, snapshot:1659927798431036|[end_task] u=103 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927798506428|[calc_checksum_by_commit] u=17 ret:0, checksum:579974452, checksum_log_ts:9223372036854775807|[submit_log] u=31 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=33103 trans_version:1659927798505127, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:988473, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=33302) [2022-08-08 11:03:18.633771] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=12043) [2022-08-08 11:03:18.641868] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20953, time_dist: generate_cluster_version=20949) [2022-08-08 11:03:18.860772] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:18.863849] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:18.863969] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:19.530330] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:19.530367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:19.530897] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:19.530932] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667526, retry interval=1000000, ret=-4179) [2022-08-08 11:03:19.556807] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:19.556847] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:19.559008] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:19.559046] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667521, retry interval=1000000, ret=-4179) [2022-08-08 11:03:19.623233] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12241] [dc=0] clock out of order(cur_ts=1659927799623226, cur_ts_=1659927799589372, delta=33854) [2022-08-08 11:03:19.723187] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=40411) [2022-08-08 11:03:19.723743] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=42682, time_dist: do_cluster_heartbeat=4, generate_cluster_version=42675) [2022-08-08 11:03:19.724342] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19494, time_dist: generate_cluster_version=19492) [2022-08-08 11:03:19.818287] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:03:19.818326] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:19.866682] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:19.871250] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:19.871279] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:20.059168] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927800059162, cur_ts_=1659927800038417, delta=20745) [2022-08-08 11:03:20.112766] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14972704}) [2022-08-08 11:03:20.302480] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927798356216, end_time=1659927800302469, elapsed_time=1946253, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:03:20.336814] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:20.516548] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.516566] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.516572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.516576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.516766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.516808] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.517063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3359-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:20.533846] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B3B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:20.558796] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:20.558848] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667525, retry interval=1000000, ret=-4179) [2022-08-08 11:03:20.570615] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11124) [2022-08-08 11:03:20.570929] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10630][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511627916, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:988477, proposal_id:{time_to_usec:1659815620310360, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:53998311, size:639}, after_consume_timestamp:1659927800558093}, end_time=1659927800570686, begin_time=1659927800558093, cost_time=12593) [2022-08-08 11:03:20.577596] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:20.577661] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667520, retry interval=1000000, ret=-4179) [2022-08-08 11:03:20.873656] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:20.877372] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:20.877398] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:21.206642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.243830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.382564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.514071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:21.552739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:21.578858] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:21.581085] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=209] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:21.581158] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667524, retry interval=1000000, ret=-4179) [2022-08-08 11:03:21.598090] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:21.599521] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:21.599774] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=204] [dc=0] task process failed, start retry(max retry time=9223372036854667519, retry interval=1000000, ret=-4179) [2022-08-08 11:03:21.604499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:21.640231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.650432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.666483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.667311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:21.677893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:21.769183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:21.770641] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:21.779673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:21.880676] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:21.883959] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:21.883995] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:21.904962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:21.936262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:22.034042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:22.047315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:22.115469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:22.604292] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6307-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:22.604572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6307-0-0] [lt=258] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:22.606814] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:22.606862] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667523, retry interval=1000000, ret=-4179) [2022-08-08 11:03:22.622886] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:22.623102] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15532) [2022-08-08 11:03:22.625122] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:22.625191] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667518, retry interval=1000000, ret=-4179) [2022-08-08 11:03:22.886649] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:22.889977] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:22.890044] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:23.545979] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927803545973, cur_ts_=1659927803525548, delta=20425) [2022-08-08 11:03:23.624422] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=19679) [2022-08-08 11:03:23.631593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4C7F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:23.631622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4C7F-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:23.635284] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31219, time_dist: do_cluster_heartbeat=7, generate_cluster_version=31209) [2022-08-08 11:03:23.638874] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:23.638907] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667522, retry interval=1000000, ret=-4179) [2022-08-08 11:03:23.652211] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12297) [2022-08-08 11:03:23.664261] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:23.664300] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667517, retry interval=1000000, ret=-4179) [2022-08-08 11:03:23.665631] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11112, time_dist: generate_cluster_version=11108) [2022-08-08 11:03:23.781167] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927803781161, cur_ts_=1659927803750932, delta=30229) [2022-08-08 11:03:23.893665] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:23.898443] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:23.898536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:24.027749] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18039) [2022-08-08 11:03:24.027973] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19029, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19016) [2022-08-08 11:03:24.640860] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B41-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:24.641041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B41-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:24.659264] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:24.659316] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667521, retry interval=1000000, ret=-4179) [2022-08-08 11:03:24.683539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:24.683873] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=175] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:24.685351] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:24.685404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667516, retry interval=1000000, ret=-4179) [2022-08-08 11:03:24.877880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:24.913480] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:24.922378] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:24.922404] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:24.984730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=110] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.039353] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11615) [2022-08-08 11:03:25.039718] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12715, time_dist: generate_cluster_version=12713) [2022-08-08 11:03:25.040533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.049102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.203695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.204901] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=404] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.212045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:25.287226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.287256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=140] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.297577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:25.337520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.338561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:25.355903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:25.370818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.402532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:25.402619] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.405500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.411431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:25.411911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.412016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.419365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.432643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:25.457563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:25.484938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.496651] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.497323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.500687] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.507194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=164] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.521550] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.538569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.538675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.538675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.538789] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.542376] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.548909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.549021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.559300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.590179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:25.590254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.603220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.642112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.652586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.661559] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4C82-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:25.680430] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:25.682997] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:25.683052] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667520, retry interval=1000000, ret=-4179) [2022-08-08 11:03:25.683147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:25.715846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:25.716742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.716873] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927805716870, cur_ts_=1659927805685524, delta=31346) [2022-08-08 11:03:25.724913] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:25.726761] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:25.726803] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667515, retry interval=1000000, ret=-4179) [2022-08-08 11:03:25.727025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:25.727284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:25.778669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:25.788500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.851031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.861324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:25.925780] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:25.929161] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:25.929222] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:25.943818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:26.007792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:26.038575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:26.061332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:26.310080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:26.592805] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] clock out of order(cur_ts=1659927806592798, cur_ts_=1659927806554129, delta=38669) [2022-08-08 11:03:26.701621] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:26.703142] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:26.703205] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667519, retry interval=1000000, ret=-4179) [2022-08-08 11:03:26.750800] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:26.750866] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667514, retry interval=1000000, ret=-4179) [2022-08-08 11:03:26.934179] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:26.937274] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:26.937310] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:27.694076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:27.724391] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:27.724453] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667518, retry interval=1000000, ret=-4179) [2022-08-08 11:03:27.797599] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:27.797711] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667513, retry interval=1000000, ret=-4179) [2022-08-08 11:03:27.940034] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:27.943271] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:27.943333] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:28.441012] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=155] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15668) [2022-08-08 11:03:28.441218] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16661, time_dist: generate_cluster_version=16659) [2022-08-08 11:03:28.462437] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19144) [2022-08-08 11:03:28.462725] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20207, time_dist: generate_cluster_version=20205) [2022-08-08 11:03:28.655791] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19806) [2022-08-08 11:03:28.655997] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20941, time_dist: do_cluster_heartbeat=7, generate_cluster_version=20932) [2022-08-08 11:03:28.676899] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20637) [2022-08-08 11:03:28.677171] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21600, time_dist: generate_cluster_version=21598) [2022-08-08 11:03:28.757885] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:28.757937] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667517, retry interval=1000000, ret=-4179) [2022-08-08 11:03:28.778849] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19388) [2022-08-08 11:03:28.824056] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:28.824175] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667512, retry interval=1000000, ret=-4179) [2022-08-08 11:03:28.855865] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11433) [2022-08-08 11:03:28.856089] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12521, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12510) [2022-08-08 11:03:28.887202] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11934) [2022-08-08 11:03:28.887434] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12944, time_dist: generate_cluster_version=12942) [2022-08-08 11:03:28.919840] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11210) [2022-08-08 11:03:28.920068] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12179, time_dist: generate_cluster_version=12177) [2022-08-08 11:03:28.945998] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:28.950555] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:28.950586] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:29.008722] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927809008711, cur_ts_=1659927808987578, delta=21133) [2022-08-08 11:03:29.072959] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14908, time_dist: generate_cluster_version=14903) [2022-08-08 11:03:29.109200] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12632, time_dist: generate_cluster_version=12628) [2022-08-08 11:03:29.163827] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927809163819, cur_ts_=1659927809137161, delta=26658) [2022-08-08 11:03:29.660806] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12655) [2022-08-08 11:03:29.661022] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13625, time_dist: do_cluster_heartbeat=10, generate_cluster_version=13613) [2022-08-08 11:03:29.693993] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11411) [2022-08-08 11:03:29.694181] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14082, time_dist: generate_cluster_version=14080) [2022-08-08 11:03:29.775601] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:29.775644] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667516, retry interval=1000000, ret=-4179) [2022-08-08 11:03:29.786715] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10365) [2022-08-08 11:03:29.819176] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:03:29.819206] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:29.860290] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:29.860334] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667511, retry interval=1000000, ret=-4179) [2022-08-08 11:03:29.958776] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:29.963619] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:29.963686] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:30.028940] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14977088}) [2022-08-08 11:03:30.221794] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927808357223, end_time=1659927810221781, elapsed_time=1864558, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:03:30.393380] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:30.778648] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6313-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:30.797009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:30.798635] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:30.798686] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667515, retry interval=1000000, ret=-4179) [2022-08-08 11:03:30.879172] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:30.880643] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:30.880685] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667510, retry interval=1000000, ret=-4179) [2022-08-08 11:03:30.966699] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:30.969827] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:30.969878] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:31.224090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.268377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.400813] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.530173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:31.573028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:31.625170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:31.662082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.666185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=566] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.684085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.687922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=158] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:31.694391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:31.787572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:31.790536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:31.803438] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:31.818264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:31.819949] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:31.819999] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667514, retry interval=1000000, ret=-4179) [2022-08-08 11:03:31.901548] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:31.901649] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667509, retry interval=1000000, ret=-4179) [2022-08-08 11:03:31.925519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:31.958731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:31.972040] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:31.975300] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:31.975390] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:32.051016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:32.067930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:32.131935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:32.690286] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:32.821811] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B4D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:32.840619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:32.841861] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:32.841924] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667513, retry interval=1000000, ret=-4179) [2022-08-08 11:03:32.924631] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:32.925796] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:32.925966] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667508, retry interval=1000000, ret=-4179) [2022-08-08 11:03:32.980465] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:32.983535] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:32.983573] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:33.861623] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:33.861687] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667512, retry interval=1000000, ret=-4179) [2022-08-08 11:03:33.943281] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:33.945441] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:33.945503] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667507, retry interval=1000000, ret=-4179) [2022-08-08 11:03:33.986109] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:33.989609] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:33.989673] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:34.864367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6319-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:34.886187] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:34.887713] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:34.887890] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] task process failed, start retry(max retry time=9223372036854667511, retry interval=1000000, ret=-4179) [2022-08-08 11:03:34.894671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:34.947505] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B50-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:34.965901] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:34.968138] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:34.968252] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667506, retry interval=1000000, ret=-4179) [2022-08-08 11:03:34.992017] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:34.996899] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:34.996988] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:35.005949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.062462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.064520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.223134] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.223378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.231836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:35.302593] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.302678] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.322445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:35.360164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:35.362534] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.378468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:35.393818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.421083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:35.427058] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.427602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.427667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.430743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.433589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:35.434954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.454272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:35.477352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:35.502436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.520053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.520095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.520106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.530282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.544042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.559031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.560116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.560150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.560212] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.565372] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.570347] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.571873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.576033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.611607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:35.613656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.630283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.666864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.672095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.708333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:35.737097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:35.739627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.750033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:35.750174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:35.803068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:35.813476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.872437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.884031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:35.912181] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=205] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:35.912252] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667510, retry interval=1000000, ret=-4179) [2022-08-08 11:03:35.964317] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:35.989230] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:35.989290] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667505, retry interval=1000000, ret=-4179) [2022-08-08 11:03:36.000478] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:36.004270] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:36.004367] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:36.033137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:36.063491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:36.079020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:36.253901] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:03:36.260108] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927816260103, cur_ts_=1659927816237473, delta=22630) [2022-08-08 11:03:36.331961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:36.658696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:36.658911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:36.659069] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:36.941912] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=192] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:36.941982] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667509, retry interval=1000000, ret=-4179) [2022-08-08 11:03:37.007123] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:37.010397] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:37.010422] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:37.017080] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:37.017195] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667504, retry interval=1000000, ret=-4179) [2022-08-08 11:03:37.714089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:37.972067] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:37.972117] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:37.973432] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:37.973497] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667508, retry interval=1000000, ret=-4179) [2022-08-08 11:03:38.015757] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:38.021193] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:38.021217] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:38.039738] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:38.039833] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667503, retry interval=1000000, ret=-4179) [2022-08-08 11:03:38.998732] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:38.998768] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667507, retry interval=1000000, ret=-4179) [2022-08-08 11:03:39.025532] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:39.029056] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:39.029093] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:39.062737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:39.062760] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:39.064258] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:39.064296] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667502, retry interval=1000000, ret=-4179) [2022-08-08 11:03:39.310037] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927819310029, cur_ts_=1659927819288373, delta=21656) [2022-08-08 11:03:39.317377] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=28223) [2022-08-08 11:03:39.319161] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30670, time_dist: generate_cluster_version=30666) [2022-08-08 11:03:39.518522] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35298, time_dist: generate_cluster_version=35293) [2022-08-08 11:03:39.659309] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] [trans sampling] (TRACE=begin_ts=1659927819641967 2022-08-08 03:03:39.641967|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:15920708909806673601, inc:84109620, addr:{ip:"192.168.11.128", port:2882}, t:1659927819640000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=40 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC5B1-0-0, start_time:1659927819641850, snapshot:1659927819627478|[end_task] u=139 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927819642159|[calc_checksum_by_commit] u=19 ret:0, checksum:1288617084, checksum_log_ts:9223372036854775807|[submit_log] u=28 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=17005 trans_version:1659927819642095, uref:1073741825|[exiting] u=34 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8317506, logging:0, pending:0, uref:1|[end_trans_cb] u=6 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=17295) [2022-08-08 11:03:39.661127] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19653, time_dist: do_cluster_heartbeat=48, generate_cluster_version=19603) [2022-08-08 11:03:39.819718] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:03:39.819739] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:39.986337] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14981536}) [2022-08-08 11:03:40.030289] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:40.030606] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:40.032039] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:40.032076] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667506, retry interval=1000000, ret=-4179) [2022-08-08 11:03:40.032165] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:40.036227] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:40.036255] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:40.082902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:40.083082] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=164] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:40.084375] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:40.084505] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667501, retry interval=1000000, ret=-4179) [2022-08-08 11:03:40.185135] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927818365870, end_time=1659927820185123, elapsed_time=1819253, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:03:40.362292] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927820362284, cur_ts_=1659927820338214, delta=24070) [2022-08-08 11:03:40.455278] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:41.038771] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:41.046184] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:41.046208] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:41.058431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:41.058452] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:41.060119] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:41.060335] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667505, retry interval=1000000, ret=-4179) [2022-08-08 11:03:41.105545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:41.107039] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:41.107120] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667500, retry interval=1000000, ret=-4179) [2022-08-08 11:03:41.249086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.285791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.426033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.551719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:41.592323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:41.642726] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:41.682992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.690005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.704473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.711828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:41.716900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:41.808570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:41.810592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:41.820484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:41.945213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:41.974731] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:42.048877] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:42.069131] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927822069109, cur_ts_=1659927822043314, delta=25795) [2022-08-08 11:03:42.072101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:42.078971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:42.078994] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:42.114456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:42.117921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.117949] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B1C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121135] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B1C-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B1C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121164] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B1C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121169] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B1C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121191] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B1C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.121319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.142028] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=1561] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14810) [2022-08-08 11:03:42.144005] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17614, time_dist: generate_cluster_version=17611) [2022-08-08 11:03:42.151276] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:42.151315] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667499, retry interval=1000000, ret=-4179) [2022-08-08 11:03:42.153324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:42.154821] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.155137] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:42.156455] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:42.156491] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667504, retry interval=1000000, ret=-4179) [2022-08-08 11:03:42.477707] WARN [COMMON] ObBucketTryRLockAllGuard (ob_bucket_lock.h:240) [10914][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to try read lock all buckets(ret=-4023) [2022-08-08 11:03:42.477735] WARN [STORAGE] get_create_schema_version (ob_pg_storage.cpp:1484) [10914][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] get max pg schema version rdlock error(ret=-4023, pkey_={tid:1099511628013, partition_id:0, part_cnt:0}) [2022-08-08 11:03:42.495082] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14294) [2022-08-08 11:03:42.495839] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16271, time_dist: generate_cluster_version=16268) [2022-08-08 11:03:42.654077] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927822653265 2022-08-08 03:03:42.653265|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:2904721057844392316, inc:84109738, addr:{ip:"192.168.11.128", port:2882}, t:1659927822651000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC5C0-0-0, start_time:1659927822653127, snapshot:1659927822638641|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927822653407|[calc_checksum_by_commit] u=17 ret:0, checksum:919850647, checksum_log_ts:9223372036854775807|[submit_log] u=69 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=498 trans_version:1659927822651289, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8317521, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=773) [2022-08-08 11:03:43.122026] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:43.125308] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:43.125344] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:43.176342] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:43.176378] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667498, retry interval=1000000, ret=-4179) [2022-08-08 11:03:43.182429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:43.182507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:43.184699] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:43.184736] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667503, retry interval=1000000, ret=-4179) [2022-08-08 11:03:43.959274] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] clock out of order(cur_ts=1659927823959269, cur_ts_=1659927823935911, delta=23358) [2022-08-08 11:03:44.142312] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:44.143520] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927824143515, cur_ts_=1659927824117567, delta=25948) [2022-08-08 11:03:44.160818] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:44.160844] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:44.177905] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34450) [2022-08-08 11:03:44.178116] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35694, time_dist: do_cluster_heartbeat=12, generate_cluster_version=35680) [2022-08-08 11:03:44.260573] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:44.260879] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854667497, retry interval=1000000, ret=-4179) [2022-08-08 11:03:44.263865] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:44.263897] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667502, retry interval=1000000, ret=-4179) [2022-08-08 11:03:44.710882] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:44.914273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:45.030636] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.084694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.088767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.178941] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:45.183427] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:45.183467] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:45.247497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.248998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.249514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:45.289120] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:45.289153] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667501, retry interval=1000000, ret=-4179) [2022-08-08 11:03:45.289876] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:45.289900] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667496, retry interval=1000000, ret=-4179) [2022-08-08 11:03:45.324267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.324898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.341714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:45.377211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:45.387800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.398055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:45.410547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.437325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:45.443387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.452002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.452772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.457669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.457813] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:45.463846] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.474103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:45.502967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:45.525408] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.535727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.538060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.542958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.545966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.566306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.576865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.576939] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.576971] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.578709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.585866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.587655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.612491] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659927825612484, cur_ts_=1659927825587312, delta=25172) [2022-08-08 11:03:45.622262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.623083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.632736] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.633511] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:45.653784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.684675] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.695254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.726336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:45.757888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:45.760545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.768052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:45.770946] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:45.820998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:45.835379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.892391] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.902765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:45.985152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=217] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:46.056359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:46.080852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:46.103124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:46.185993] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:46.189428] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:46.189467] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:46.299861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:46.300053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:46.317710] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:46.317969] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:46.320666] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=189] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:46.320890] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] task process failed, start retry(max retry time=9223372036854667500, retry interval=1000000, ret=-4179) [2022-08-08 11:03:46.327484] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12646) [2022-08-08 11:03:46.328510] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19865, time_dist: do_cluster_heartbeat=9, generate_cluster_version=19854) [2022-08-08 11:03:46.331309] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:46.331439] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:46.332659] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:46.332707] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667495, retry interval=1000000, ret=-4179) [2022-08-08 11:03:46.355226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:46.725362] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33A7-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.725389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33A7-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.725604] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33A7-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.725789] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33A7-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.726138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.726529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.726560] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:46.726577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:47.125083] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2462] [dc=0] clock out of order(cur_ts=1659927827125078, cur_ts_=1659927827100265, delta=24813) [2022-08-08 11:03:47.192085] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:47.195393] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:47.195499] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:47.335284] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.335538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.335579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.335980] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.336002] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.336013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.336431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.336464] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B62-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.365830] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.366397] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18766) [2022-08-08 11:03:47.367104] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:47.367138] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667499, retry interval=1000000, ret=-4179) [2022-08-08 11:03:47.367749] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22224, time_dist: generate_cluster_version=22221) [2022-08-08 11:03:47.372114] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:47.372152] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667494, retry interval=1000000, ret=-4179) [2022-08-08 11:03:47.492331] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86445-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.492704] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86445-0-0] [lt=356] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.504240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.504267] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.514749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.515014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.518782] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.518917] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.522783] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.523009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.686662] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC8653F-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.686792] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.686819] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.687532] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.687565] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.735003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:47.893717] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC8665F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.893746] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC8665F-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.893756] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC8665F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.893829] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC8665F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:47.893929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:48.064665] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=258] [dc=0] clock out of order(cur_ts=1659927828064661, cur_ts_=1659927828029091, delta=35570) [2022-08-08 11:03:48.198259] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:48.201872] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:48.201906] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:48.389400] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927828389397, cur_ts_=1659927828364853, delta=24544) [2022-08-08 11:03:48.406748] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CA4-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:48.406835] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CA4-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:48.457642] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:48.459676] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667493, retry interval=1000000, ret=-4179) [2022-08-08 11:03:48.479364] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=219] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:48.479473] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667498, retry interval=1000000, ret=-4179) [2022-08-08 11:03:48.481908] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15134) [2022-08-08 11:03:48.687390] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927828687385, cur_ts_=1659927828667071, delta=20314) [2022-08-08 11:03:48.933994] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927828933989, cur_ts_=1659927828910939, delta=23050) [2022-08-08 11:03:49.022833] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15109) [2022-08-08 11:03:49.023039] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23207, time_dist: generate_cluster_version=23205) [2022-08-08 11:03:49.048324] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927829048320, cur_ts_=1659927829028166, delta=20154) [2022-08-08 11:03:49.168239] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] clock out of order(cur_ts=1659927829168233, cur_ts_=1659927829144301, delta=23932) [2022-08-08 11:03:49.203222] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20251) [2022-08-08 11:03:49.203431] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23964, time_dist: generate_cluster_version=23962) [2022-08-08 11:03:49.204321] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:49.214290] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:49.214685] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=379] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:49.278676] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927829278669, cur_ts_=1659927829248384, delta=30285) [2022-08-08 11:03:49.363942] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12463, time_dist: do_cluster_heartbeat=7, generate_cluster_version=12452) [2022-08-08 11:03:49.368202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86D79-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.368411] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86D79-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.368639] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86D79-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.369122] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC86D79-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.369551] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.369579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.369593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.372354] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18244, time_dist: do_cluster_heartbeat=7, generate_cluster_version=18233) [2022-08-08 11:03:49.459200] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10495][0][YB42C0A80B80-0005E3CFCBC86DE5-0-0] [lt=2] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1752 is_safe_read=false read_snapshot=1659927829454907 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659927829456477 abs_expired_time=1659927831388629 stmt_timeout=1932152 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21597259 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659927829384940 start_version=-1 trans_version=1659927829454867 commit_version=0 stmt_start_time=1659927829453751 abs_expired_time=1659927859353912 stmt_timeout=29900161 abs_lock_wait_timeout=1659927859353912 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:17387781678843305772, inc:84112449, addr:"192.168.11.128:2882", t:1659927829453751} checksum_log_ts=9223372036854775807, version=1659927829454907) [2022-08-08 11:03:49.518686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.518705] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.539819] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927829539813, cur_ts_=1659927829517932, delta=21881) [2022-08-08 11:03:49.554042] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:49.554074] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667497, retry interval=1000000, ret=-4179) [2022-08-08 11:03:49.576509] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=193] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:49.577605] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] task process failed, start retry(max retry time=9223372036854667492, retry interval=1000000, ret=-4179) [2022-08-08 11:03:49.599423] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.607979] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.608247] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.611982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.612189] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.615547] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.615694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.624692] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.624880] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.713765] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.713785] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.761185] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10231, time_dist: do_cluster_heartbeat=7, generate_cluster_version=10222) [2022-08-08 11:03:49.764603] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12983, time_dist: print_stat=5, do_cluster_heartbeat=2, generate_cluster_version=12970, cluster_service_self_check=4) [2022-08-08 11:03:49.765798] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.765970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:49.811534] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=249] [dc=0] clock out of order(cur_ts=1659927829811528, cur_ts_=1659927829791073, delta=20455) [2022-08-08 11:03:49.834804] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:03:49.834850] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:50.006228] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927830006221, cur_ts_=1659927829984647, delta=21574) [2022-08-08 11:03:50.179324] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14985856}) [2022-08-08 11:03:50.189809] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16466, time_dist: do_cluster_heartbeat=7, generate_cluster_version=16457) [2022-08-08 11:03:50.209256] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27356, time_dist: do_cluster_heartbeat=8, generate_cluster_version=27345) [2022-08-08 11:03:50.218192] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:50.235260] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:50.235284] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:50.396902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.397114] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.397252] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.397521] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.406663] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927828364858, end_time=1659927830406654, elapsed_time=2041796, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:03:50.443669] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927830443664, cur_ts_=1659927830414360, delta=29304) [2022-08-08 11:03:50.530678] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:50.591224] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:50.591258] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667496, retry interval=1000000, ret=-4179) [2022-08-08 11:03:50.598002] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:50.598033] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667491, retry interval=1000000, ret=-4179) [2022-08-08 11:03:50.608615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC872AD-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.608670] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC872AD-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.608823] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC872AD-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.608923] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC872AD-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.609933] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.793644] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.793676] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.793886] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.793920] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.816708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.816751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.820314] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.820457] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:50.820595] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10495][0][YB42C0A80B80-0005E3CFCBC87361-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:51.248648] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:51.252117] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:51.252143] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:51.274827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.307744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.442342] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.547269] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=286] [dc=0] clock out of order(cur_ts=1659927831547263, cur_ts_=1659927831513022, delta=34241) [2022-08-08 11:03:51.569650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:51.613999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:51.640228] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:51.640285] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667490, retry interval=1000000, ret=-4179) [2022-08-08 11:03:51.666892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:51.667041] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927831667036, cur_ts_=1659927831623092, delta=43944) [2022-08-08 11:03:51.668903] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:51.668937] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667495, retry interval=1000000, ret=-4179) [2022-08-08 11:03:51.673691] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21022) [2022-08-08 11:03:51.678327] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] [trans sampling] (TRACE=begin_ts=1659927831675077 2022-08-08 03:03:51.675077|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:4821402319852870033, inc:84113978, addr:{ip:"192.168.11.128", port:2882}, t:1659927831673000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F7FB1-0-0, start_time:1659927831674961, snapshot:1659927831652883|[end_task] u=229 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927831675325|[calc_checksum_by_commit] u=19 ret:0, checksum:900124922, checksum_log_ts:9223372036854775807|[submit_log] u=47 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=2842 trans_version:1659927831673493, uref:1073741825|[exiting] u=22 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:988538, logging:0, pending:0, uref:1|[end_trans_cb] u=18 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=3207) [2022-08-08 11:03:51.708100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.708160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.720356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.730516] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:51.740706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:51.760883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33B8-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:51.827585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:51.834117] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:51.842829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:51.970918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:51.995825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:52.088929] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:52.132188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:52.173736] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:52.255288] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:52.258998] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:52.259023] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:52.645966] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CAB-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:52.646471] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CAB-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:52.668978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:52.670764] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:52.670799] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667489, retry interval=1000000, ret=-4179) [2022-08-08 11:03:52.690575] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:52.717355] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927832717349, cur_ts_=1659927832689849, delta=27500) [2022-08-08 11:03:52.720020] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:52.720063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:52.721902] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:52.721951] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667494, retry interval=1000000, ret=-4179) [2022-08-08 11:03:52.764790] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33BB-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:52.764824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33BB-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:53.263434] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:53.269016] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:53.269058] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:53.673282] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B6E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:53.673350] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B6E-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:53.673550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B6E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:53.693908] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:53.695171] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:53.695221] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667488, retry interval=1000000, ret=-4179) [2022-08-08 11:03:53.742157] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:53.742450] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:53.743723] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:53.743783] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667493, retry interval=1000000, ret=-4179) [2022-08-08 11:03:53.772325] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33BE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:53.772458] WARN [COMMON] on_full_return (ob_io_struct.cpp:2546) [10864][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] push io request into callback queue failed(ret=-4072, req={is_inited_:true, is_finished_:true, is_canceled_:true, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:488}, offset_:774144, size_:36864, flag_:{mode_:0, category_:1, wait_event_id_:1, reserved_:0}, write_buf_:null, callback_:0x7f68e739c770}, deadline_ts_:1659927833772054, control_block_:0x7f6833bc4560, raw_buf_:null, io_buf_:0x7f67e3038000, io_offset_:774144, io_size_:36864, complete_size_:36864, time_log_:{begin_ts_:1659927833772053, enqueue_ts_:1659927833772054, dequeue_ts_:1659927833772193, submit_ts_:1659927833772196, return_ts_:1659927833772436, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1659927833772437}, channel_:0x7f69a4ef6310, ref_cnt_:2, out_ref_cnt_:0, trace_id_:YB42C0A80B80-0005E3CFCABE33BE-0-0, ret_code_:{io_ret_:-4072, fs_errno_:0}, retry_count_:0, copied_callback_:0x7f680b7f3d60}) [2022-08-08 11:03:53.772498] WARN [COMMON] get_events (ob_io_struct.cpp:2500) [10864][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] process full return io request failed(ret=-4072, *req={is_inited_:true, is_finished_:true, is_canceled_:true, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:488}, offset_:774144, size_:36864, flag_:{mode_:0, category_:1, wait_event_id_:1, reserved_:0}, write_buf_:null, callback_:0x7f68e739c770}, deadline_ts_:1659927833772054, control_block_:0x7f6833bc4560, raw_buf_:null, io_buf_:0x7f67e3038000, io_offset_:774144, io_size_:36864, complete_size_:36864, time_log_:{begin_ts_:1659927833772053, enqueue_ts_:1659927833772054, dequeue_ts_:1659927833772193, submit_ts_:1659927833772196, return_ts_:1659927833772436, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1659927833772437}, channel_:0x7f69a4ef6310, ref_cnt_:1, out_ref_cnt_:0, trace_id_:YB42C0A80B80-0005E3CFCABE33BE-0-0, ret_code_:{io_ret_:-4072, fs_errno_:0}, retry_count_:0, copied_callback_:0x7f680b7f3d60}) [2022-08-08 11:03:53.775372] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:53.999999] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13927, time_dist: do_cluster_heartbeat=8, generate_cluster_version=13915) [2022-08-08 11:03:54.271248] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:54.276879] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:54.276922] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:54.698279] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B2F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:54.718310] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:54.718415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:54.720091] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:54.720165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667487, retry interval=1000000, ret=-4179) [2022-08-08 11:03:54.750079] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:54.767843] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:54.767929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:54.769333] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:54.769383] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667492, retry interval=1000000, ret=-4179) [2022-08-08 11:03:54.779414] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33C1-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:54.779487] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33C1-0-0] [lt=59] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:54.779971] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:54.780103] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:54.780249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:54.780435] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:54.937972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:55.055198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.106230] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.106269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.265132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:55.266401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.268278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.279130] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:55.282334] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:55.282382] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:55.319412] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927835319405, cur_ts_=1659927835293536, delta=25869) [2022-08-08 11:03:55.340293] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.346299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.363514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:55.402264] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:55.408964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.415737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:55.429086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.459867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:55.460182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=175] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.470141] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.470552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=325] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.480843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.480892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:55.481484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.491524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:03:55.522413] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:55.543400] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.553567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.553950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.563879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.564537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.591180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.594239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.595335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.595793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.601819] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.606179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25234] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.606590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.643415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.643877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.650398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:55.652225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=155] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.673061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.699890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.710366] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.723022] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC633A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:55.756822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:55.757074] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927835757070, cur_ts_=1659927835728221, delta=28849) [2022-08-08 11:03:55.777357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:03:55.778459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.779533] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:55.779979] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:55.781024] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:55.781056] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667486, retry interval=1000000, ret=-4179) [2022-08-08 11:03:55.788268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:55.788355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:03:55.796377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:55.796441] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:55.798319] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:55.798353] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667491, retry interval=1000000, ret=-4179) [2022-08-08 11:03:55.840425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:55.860585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.919707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:55.923086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:56.006419] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:03:56.072349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:56.104244] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:03:56.118010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:56.282341] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:03:56.285576] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:56.288952] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:56.288990] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:56.375943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:03:56.711846] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] clock out of order(cur_ts=1659927836711839, cur_ts_=1659927836684863, delta=26976) [2022-08-08 11:03:56.802143] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:56.802162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:56.804120] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:56.804160] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667485, retry interval=1000000, ret=-4179) [2022-08-08 11:03:56.817994] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:56.818250] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:56.819950] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:56.819983] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667490, retry interval=1000000, ret=-4179) [2022-08-08 11:03:57.292659] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:57.296222] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:57.296260] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:57.755283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:03:57.785981] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] [trans sampling] (TRACE=begin_ts=1659927837785095 2022-08-08 03:03:57.785095|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:12786443477406720694, inc:84114199, addr:{ip:"192.168.11.128", port:2882}, t:1659927837785000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC60B-0-0, start_time:1659927837784917, snapshot:1659927837718151|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659927837785236|[calc_checksum_by_commit] u=18 ret:0, checksum:1211620586, checksum_log_ts:9223372036854775807|[submit_log] u=48 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=545 trans_version:1659927837785051, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8317596, logging:0, pending:0, uref:1|[end_trans_cb] u=42 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=824) [2022-08-08 11:03:57.837478] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:57.837957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=461] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:57.841168] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:57.841232] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667484, retry interval=1000000, ret=-4179) [2022-08-08 11:03:57.842861] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:03:57.844232] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:57.844268] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667489, retry interval=1000000, ret=-4179) [2022-08-08 11:03:58.298387] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:58.301863] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:58.301898] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:58.820114] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33CD-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:58.820133] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33CD-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:58.877461] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:58.877495] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667488, retry interval=1000000, ret=-4179) [2022-08-08 11:03:58.889941] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:58.890005] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667483, retry interval=1000000, ret=-4179) [2022-08-08 11:03:59.070345] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E72F5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.070398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][YB42C0A80B80-0005E3CFC47E72F5-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.111182] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] clock out of order(cur_ts=1659927839111175, cur_ts_=1659927839091131, delta=20044) [2022-08-08 11:03:59.131377] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.131395] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.153587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.153606] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.158435] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.158455] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.164017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.164244] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=212] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.207242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.207276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.214774] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.214805] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.220916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.220950] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.225347] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.225365] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.282482] WARN [SERVER] load_info_item (ob_backup_operator.cpp:1396) [10495][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] backup info is not exist yet, wait later(ret=-9032, tenant_id=1, sql=SELECT name, value FROM __all_tenant_backup_info WHERE name = 'tenant_name_backup_schema_version' AND tenant_id = 1) [2022-08-08 11:03:59.282531] WARN [SERVER] get_tenant_name_backup_schema_version (ob_backup_operator.cpp:1487) [10495][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] tenant backup info not exist(ret=-9032, backup_schema_version=0) [2022-08-08 11:03:59.296191] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.296268] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.306686] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:03:59.310873] WARN [SERVER] load_info_item (ob_backup_operator.cpp:1396) [10495][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] backup info is not exist yet, wait later(ret=-9032, tenant_id=1001, sql=SELECT name, value FROM __all_tenant_backup_info WHERE name = 'backup_schema_version' AND tenant_id = 1001) [2022-08-08 11:03:59.311026] WARN [SERVER] get_backup_schema_version (ob_backup_operator.cpp:1462) [10495][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] tenant backup info not exist(ret=-9032, backup_schema_version=0) [2022-08-08 11:03:59.325483] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:03:59.325513] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:03:59.335907] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.335967] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.387337] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10291, time_dist: generate_cluster_version=10287) [2022-08-08 11:03:59.410252] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927839410247, cur_ts_=1659927839387346, delta=22901) [2022-08-08 11:03:59.773056] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.789818] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.789852] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.790674] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.790695] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.804431] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.818739] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10117, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10105) [2022-08-08 11:03:59.833618] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:03:59.833998] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=370] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:03:59.840728] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.840789] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.841453] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.841485] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.880190] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927839880184, cur_ts_=1659927839850171, delta=30013) [2022-08-08 11:03:59.957582] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12512) [2022-08-08 11:03:59.965803] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21795, time_dist: do_cluster_heartbeat=5, generate_cluster_version=21787) [2022-08-08 11:03:59.971913] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:59.971965] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667482, retry interval=1000000, ret=-4179) [2022-08-08 11:03:59.976470] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:03:59.976501] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667487, retry interval=1000000, ret=-4179) [2022-08-08 11:03:59.985795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.985814] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.985820] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.985824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:03:59.985827] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.000739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.000759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.000764] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.000768] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.000772] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.032758] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29103) [2022-08-08 11:04:00.034154] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31191, time_dist: generate_cluster_version=31188) [2022-08-08 11:04:00.034465] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.034481] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.034572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.034715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.034724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.040956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.040980] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.040986] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.040990] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.112315] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927840112302, cur_ts_=1659927840090789, delta=21513) [2022-08-08 11:04:00.138850] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.138884] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.138890] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.138893] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.138897] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.158253] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14990336}) [2022-08-08 11:04:00.185437] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13257) [2022-08-08 11:04:00.186299] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15976, time_dist: do_cluster_heartbeat=4, generate_cluster_version=15967, cluster_service_self_check=3) [2022-08-08 11:04:00.191899] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22281, time_dist: do_cluster_heartbeat=10, generate_cluster_version=22269) [2022-08-08 11:04:00.234655] WARN run1 (ob_timer.cpp:357) [10582][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase7storage20ObPartitionScheduler14MergeRetryTaskE", start_time=1659927839067407, end_time=1659927840234641, elapsed_time=1167234, this=0x7f69b1546250, thread_id=10582) [2022-08-08 11:04:00.234732] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] clock out of order(cur_ts=1659927840234727, cur_ts_=1659927840210183, delta=24544) [2022-08-08 11:04:00.327109] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:00.331213] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:00.332833] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=1603] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:00.365037] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927838365813, end_time=1659927840365026, elapsed_time=1999213, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:04:00.558966] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927840558960, cur_ts_=1659927840532086, delta=26874) [2022-08-08 11:04:00.610983] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:00.694629] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927840694616, cur_ts_=1659927840646205, delta=48411) [2022-08-08 11:04:00.803022] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=10] [dc=0] [trans sampling] (TRACE=begin_ts=1659927840801009 2022-08-08 03:04:00.801009|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:3609982631329521820, inc:84115209, addr:{ip:"192.168.11.128", port:2882}, t:1659927840799000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6848-0-0, start_time:1659927840800787, snapshot:1659927840746674|[end_task] u=148 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=11 stc:1659927840801177|[calc_checksum_by_commit] u=23 ret:0, checksum:3099487912, checksum_log_ts:9223372036854775807|[submit_log] u=316 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1329 trans_version:1659927840801202, uref:1073741825|[exiting] u=64 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=27 ret:0, log_type:64, log_id:8303014, logging:0, pending:0, uref:1|[end_trans_cb] u=36 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1972) [2022-08-08 11:04:00.831540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.831691] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.831911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.831927] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.832104] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.832262] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=122] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.832560] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D3-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:00.987222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B38-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:00.988716] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B38-0-0] [lt=1418] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:01.010151] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:01.010349] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667481, retry interval=1000000, ret=-4179) [2022-08-08 11:04:01.014835] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:01.014874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:01.016608] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:01.016744] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667486, retry interval=1000000, ret=-4179) [2022-08-08 11:04:01.101488] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927841101482, cur_ts_=1659927841079167, delta=22315) [2022-08-08 11:04:01.297026] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=154] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:01.331220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:01.339357] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:01.343245] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:01.343282] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:01.465213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:01.590665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:01.633272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=1919] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:01.691285] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:01.727201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:01.728768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:01.735488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:01.750820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:01.756271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:01.840225] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D6-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:01.840253] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D6-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:01.840269] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE33D6-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:01.841670] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:01.841698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:01.856495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:01.856784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:01.857117] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=361] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15875) [2022-08-08 11:04:01.857295] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=350] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18015, time_dist: generate_cluster_version=18013) [2022-08-08 11:04:01.865941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:01.990425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:02.012179] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:02.042822] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:02.042861] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667480, retry interval=1000000, ret=-4179) [2022-08-08 11:04:02.046046] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:02.046110] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667485, retry interval=1000000, ret=-4179) [2022-08-08 11:04:02.113953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:02.156624] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:02.196785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:02.345671] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:02.348977] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:02.349028] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:02.626029] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927842626024, cur_ts_=1659927842593533, delta=32491) [2022-08-08 11:04:02.875695] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16603) [2022-08-08 11:04:02.877836] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21141, time_dist: generate_cluster_version=21139) [2022-08-08 11:04:03.070201] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:03.070241] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667479, retry interval=1000000, ret=-4179) [2022-08-08 11:04:03.071301] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:03.071329] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667484, retry interval=1000000, ret=-4179) [2022-08-08 11:04:03.351396] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:03.355851] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:03.355910] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:04.072180] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.072254] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.072714] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.072916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.073270] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.073302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.073713] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.073737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.073982] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B7D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.096434] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:04.096488] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667483, retry interval=1000000, ret=-4179) [2022-08-08 11:04:04.099135] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.099357] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:04.104229] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:04.104326] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667478, retry interval=1000000, ret=-4179) [2022-08-08 11:04:04.358026] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:04.361841] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:04.361887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:04.963122] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:04.981283] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927844981276, cur_ts_=1659927844958929, delta=22347) [2022-08-08 11:04:05.072080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.107098] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.107264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.107313] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.107682] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.107694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.107719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.107879] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.108103] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.108127] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.108242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.108731] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.108753] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.108763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B3E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.126567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.128484] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:05.128583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.128892] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667477, retry interval=1000000, ret=-4179) [2022-08-08 11:04:05.129307] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.129402] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:05.130868] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:05.130901] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667482, retry interval=1000000, ret=-4179) [2022-08-08 11:04:05.224389] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DBF24-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:05.224599] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:05.282961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:05.283854] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.293088] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.364192] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:05.365405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.365457] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.367737] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:05.367768] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:05.380588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:05.418247] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:05.427303] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.438589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:05.449115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.475321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:05.480110] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.486205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.490202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.500142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.500225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.500319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:05.507023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:05.541645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:05.558778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.569094] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.572647] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.583146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.589712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.607196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.610338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.614322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.620820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.621757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.622010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.624758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.663256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.667146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.667709] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:05.668606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.688696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.717415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.730472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.776368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:05.805935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:05.807977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:05.808150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.808403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:05.862804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:05.876225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.951377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:05.963509] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:06.024942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:06.089788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:06.126041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:06.136091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:06.137650] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:06.159290] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:06.159331] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667481, retry interval=1000000, ret=-4179) [2022-08-08 11:04:06.165441] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:06.165551] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667476, retry interval=1000000, ret=-4179) [2022-08-08 11:04:06.179966] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:04:06.283670] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.283849] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.283911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.284029] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.284095] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.284199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.284353] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.284433] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.284613] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.289668] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.289697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.289808] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.289957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=140] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.289978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.289984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.290121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298136] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298233] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298255] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298326] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298473] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.298643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303465] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303481] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303490] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303598] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303663] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303713] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303764] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.303890] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310443] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310607] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=153] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310619] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310626] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.310630] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.315719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.315777] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.315786] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.315791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.315872] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.315905] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320098] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320104] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.320382] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.332238] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333512] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=1001] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333601] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333695] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333729] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333743] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.333749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.341898] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.342001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.342058] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.342083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.342088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.342092] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.342266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:06.375569] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:06.379149] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:06.379188] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:06.393865] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:07.198260] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:07.198296] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667480, retry interval=1000000, ret=-4179) [2022-08-08 11:04:07.203460] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:07.203584] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:07.205164] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:07.205217] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667475, retry interval=1000000, ret=-4179) [2022-08-08 11:04:07.355968] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927847355962, cur_ts_=1659927847331832, delta=24130) [2022-08-08 11:04:07.365344] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.365393] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.365442] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.365679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.365693] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.365698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.373311] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.373331] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.373337] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.373572] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.373644] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.373796] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.381381] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:07.385123] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:07.385150] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:07.402435] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.402455] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.408343] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.408358] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.408385] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.408492] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.408534] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.408716] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.419971] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:07.779250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:08.233839] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:08.233906] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667474, retry interval=1000000, ret=-4179) [2022-08-08 11:04:08.234871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:08.235065] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:08.236454] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:08.236489] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667479, retry interval=1000000, ret=-4179) [2022-08-08 11:04:08.389402] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:08.394348] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:08.394372] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:08.453066] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10974) [2022-08-08 11:04:08.458579] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21953, time_dist: generate_cluster_version=21951) [2022-08-08 11:04:09.253647] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:09.253684] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667478, retry interval=1000000, ret=-4179) [2022-08-08 11:04:09.283197] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28269) [2022-08-08 11:04:09.285743] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:09.285776] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667473, retry interval=1000000, ret=-4179) [2022-08-08 11:04:09.289635] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35548, time_dist: generate_cluster_version=35544) [2022-08-08 11:04:09.388850] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927849388844, cur_ts_=1659927849367530, delta=21314) [2022-08-08 11:04:09.397817] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:09.403659] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=108] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:09.403725] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:09.411554] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13514, time_dist: do_cluster_heartbeat=6, generate_cluster_version=13503) [2022-08-08 11:04:09.821537] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:04:09.821568] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:04:09.844638] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12395, time_dist: generate_cluster_version=12390) [2022-08-08 11:04:10.008620] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14994688}) [2022-08-08 11:04:10.025083] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12778) [2022-08-08 11:04:10.035837] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25404, time_dist: generate_cluster_version=25402) [2022-08-08 11:04:10.201760] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927848366252, end_time=1659927850201746, elapsed_time=1835494, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:04:10.255622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC634F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:10.255856] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC634F-0-0] [lt=199] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:10.266848] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12624) [2022-08-08 11:04:10.267367] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14025, time_dist: generate_cluster_version=14024) [2022-08-08 11:04:10.276476] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:10.276714] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] task process failed, start retry(max retry time=9223372036854667477, retry interval=1000000, ret=-4179) [2022-08-08 11:04:10.288824] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B86-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:10.289038] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B86-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:10.309456] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:10.311864] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:10.311918] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667472, retry interval=1000000, ret=-4179) [2022-08-08 11:04:10.405546] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:10.409026] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:10.409063] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:10.669414] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13937) [2022-08-08 11:04:10.669717] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15791, time_dist: generate_cluster_version=15789) [2022-08-08 11:04:10.685966] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:11.063550] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23794, time_dist: generate_cluster_version=23790) [2022-08-08 11:04:11.279364] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B47-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.279513] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B47-0-0] [lt=93] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.279759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B47-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.302689] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.302739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.306875] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:11.306939] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667476, retry interval=1000000, ret=-4179) [2022-08-08 11:04:11.314106] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CC6-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.314243] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CC6-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.315969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:11.331758] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.331778] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:11.332997] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:11.333190] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667471, retry interval=1000000, ret=-4179) [2022-08-08 11:04:11.346591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:11.411552] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:11.415063] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:11.415124] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:11.474751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.474811] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.474818] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.474822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.474825] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.475015] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.481928] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:11.483109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:11.607610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:11.650498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:11.714049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:11.746204] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=293] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:11.748646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=788] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:11.755702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:11.767427] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:11.776193] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:11.878226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:11.881228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:11.888292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:12.014483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:12.035871] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:12.131603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:12.179180] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:12.212385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=229] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:12.309251] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B89-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:12.325595] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:12.325701] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:12.327013] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:12.327044] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854667475, retry interval=1000000, ret=-4179) [2022-08-08 11:04:12.355822] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:12.355873] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667470, retry interval=1000000, ret=-4179) [2022-08-08 11:04:12.418297] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:12.421693] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:12.421729] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:12.487594] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:12.691049] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:13.329934] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CC9-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:13.355173] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:13.355225] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667474, retry interval=1000000, ret=-4179) [2022-08-08 11:04:13.381546] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:13.381584] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667469, retry interval=1000000, ret=-4179) [2022-08-08 11:04:13.424734] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:13.427971] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:13.428027] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:13.551401] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC07B-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:13.551471] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC07B-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:13.551743] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC07B-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:13.551848] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:13.552051] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.404822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:14.406311] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:14.406361] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667468, retry interval=1000000, ret=-4179) [2022-08-08 11:04:14.406955] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:14.407172] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:14.408327] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:14.408384] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667473, retry interval=1000000, ret=-4179) [2022-08-08 11:04:14.431845] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:14.435883] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=975] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:14.435923] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:14.623719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC0B7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.623878] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC0B7-0-0] [lt=137] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.624124] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC0B7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.624211] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC0B7-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.624312] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.624327] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.624559] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.624702] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:14.982981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:15.095258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.142741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.147055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.302948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:15.306245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.317255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.383813] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.385608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.396145] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:15.438814] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:15.439225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:15.440711] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:15.440760] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667472, retry interval=1000000, ret=-4179) [2022-08-08 11:04:15.443688] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:15.443712] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:15.449780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.451651] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:15.451719] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667467, retry interval=1000000, ret=-4179) [2022-08-08 11:04:15.460272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:15.468081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.497124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.498296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:15.501933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.515664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.520013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:15.520997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.522591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.528067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:15.564374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:15.574597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.584658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.589059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.604325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.604920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.623281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.625806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.631109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.641459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.641475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.644225] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:15.644327] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:15.646334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.646694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.686235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.688074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=232] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.688300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:15.692142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.709197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.741300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.752311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.794394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=205] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:15.825090] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:15.825182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:15.825224] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:15.833497] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.883892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:15.894185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.967474] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=1421] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:15.988822] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:16.050258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:16.105425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:16.153491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:16.153648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:16.278069] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927856278064, cur_ts_=1659927856251631, delta=26433) [2022-08-08 11:04:16.318819] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:04:16.409028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:16.447016] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:16.464914] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:16.464961] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:16.467225] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12650) [2022-08-08 11:04:16.467725] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14684, time_dist: do_cluster_heartbeat=6, generate_cluster_version=14676) [2022-08-08 11:04:16.498899] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=729] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:16.499212] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667466, retry interval=1000000, ret=-4179) [2022-08-08 11:04:16.499980] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:16.500026] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667471, retry interval=1000000, ret=-4179) [2022-08-08 11:04:16.734553] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:16.740472] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:16.772025] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.467223] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:17.470706] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:17.470761] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:17.538812] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:17.538960] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:17.544039] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:17.544822] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667470, retry interval=1000000, ret=-4179) [2022-08-08 11:04:17.562747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:17.575168] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:17.575491] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667465, retry interval=1000000, ret=-4179) [2022-08-08 11:04:17.795472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:17.833171] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC19C-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.833240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][YB42C0A80B80-0005E3CFC66DC19C-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.847265] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13765, time_dist: do_cluster_heartbeat=7, generate_cluster_version=13756) [2022-08-08 11:04:17.847395] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.855264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.855297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.855319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.868863] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.868895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.868901] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.877058] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.877091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.882135] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.882268] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=120] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.882277] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.889953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.889989] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.889996] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.895209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.895257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.895264] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.902497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.907230] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.907250] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.926019] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:17.926066] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10915][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Inappropriate ioctl for device") [2022-08-08 11:04:18.473019] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:18.476207] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:18.476234] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:18.504004] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927858504000, cur_ts_=1659927858472018, delta=31982) [2022-08-08 11:04:18.550144] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:18.550175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:18.570419] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:18.570457] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667469, retry interval=1000000, ret=-4179) [2022-08-08 11:04:18.578066] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B92-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:18.578121] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B92-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:18.626699] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927858626694, cur_ts_=1659927858596717, delta=29977) [2022-08-08 11:04:18.630341] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:18.630389] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667464, retry interval=1000000, ret=-4179) [2022-08-08 11:04:18.937098] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3409-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:18.937280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3409-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:19.504681] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:19.507687] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:19.507711] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:19.578588] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578635] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578642] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578647] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578674] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578684] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578709] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.578714] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B53-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.591228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.591249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.611881] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.612666] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:19.612839] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667468, retry interval=1000000, ret=-4179) [2022-08-08 11:04:19.653199] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] clock out of order(cur_ts=1659927859653193, cur_ts_=1659927859631247, delta=21946) [2022-08-08 11:04:19.664367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:19.666150] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=227] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:19.666658] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667463, retry interval=1000000, ret=-4179) [2022-08-08 11:04:19.822427] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:04:19.822494] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:04:20.010535] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:14999040}) [2022-08-08 11:04:20.197432] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927858367423, end_time=1659927860197418, elapsed_time=1829995, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:04:20.509165] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:20.513530] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:20.513570] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:20.645802] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:20.645858] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667467, retry interval=1000000, ret=-4179) [2022-08-08 11:04:20.682072] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927860682063, cur_ts_=1659927860656912, delta=25151) [2022-08-08 11:04:20.682216] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37137) [2022-08-08 11:04:20.683260] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=38909, time_dist: do_cluster_heartbeat=9, generate_cluster_version=38898) [2022-08-08 11:04:20.703323] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21068) [2022-08-08 11:04:20.709069] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:20.709128] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667462, retry interval=1000000, ret=-4179) [2022-08-08 11:04:20.754826] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:20.949012] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE340F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:20.949102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE340F-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:21.340385] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:21.369393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:21.511328] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927861511316, cur_ts_=1659927861473849, delta=37467) [2022-08-08 11:04:21.511761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:21.516050] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:21.520227] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:21.520272] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:21.626763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:21.671146] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:21.681716] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12187) [2022-08-08 11:04:21.682713] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17093, time_dist: do_cluster_heartbeat=11, generate_cluster_version=17078) [2022-08-08 11:04:21.694219] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25772, time_dist: do_cluster_heartbeat=8, generate_cluster_version=25762) [2022-08-08 11:04:21.697298] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:21.697504] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] task process failed, start retry(max retry time=9223372036854667466, retry interval=1000000, ret=-4179) [2022-08-08 11:04:21.741990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:21.753482] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:21.753624] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] task process failed, start retry(max retry time=9223372036854667461, retry interval=1000000, ret=-4179) [2022-08-08 11:04:21.763583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:21.774779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:21.775878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:21.784304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:21.795806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:21.900705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:21.900755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:21.911491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:22.035870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=394] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:22.059327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:22.136993] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11550, time_dist: generate_cluster_version=11542, cluster_service_self_check=4) [2022-08-08 11:04:22.148016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:22.170938] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927862170931, cur_ts_=1659927862150072, delta=20859) [2022-08-08 11:04:22.204352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:22.231816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:22.537377] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:22.540421] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:22.540458] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:22.704615] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.704645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.731581] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.731603] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.732674] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:22.732709] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667465, retry interval=1000000, ret=-4179) [2022-08-08 11:04:22.755959] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B98-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.756027] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4B98-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.773745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.773795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:22.776689] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:22.776742] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667460, retry interval=1000000, ret=-4179) [2022-08-08 11:04:22.966318] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3415-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:23.542639] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:23.546028] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:23.546081] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:23.735587] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B59-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.737957] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.738048] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.754602] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.756110] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:23.756148] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667464, retry interval=1000000, ret=-4179) [2022-08-08 11:04:23.779160] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CD8-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.779257] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CD8-0-0] [lt=58] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.800392] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.800545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:23.801828] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:23.801880] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667459, retry interval=1000000, ret=-4179) [2022-08-08 11:04:23.975866] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:23.975938] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:23.976469] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:24.549129] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:24.552408] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:24.552481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:24.784273] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:24.784336] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667463, retry interval=1000000, ret=-4179) [2022-08-08 11:04:24.820748] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:24.820887] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667458, retry interval=1000000, ret=-4179) [2022-08-08 11:04:24.983898] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE341B-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:24.984009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE341B-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:24.999219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:25.118118] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.159312] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.169358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.324752] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:25.328773] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.335245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.402661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.409713] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.421529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:25.464245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:25.474735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.481300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:25.493004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.513816] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.522705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:25.522750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.533196] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.544728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.544954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.545188] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:25.545492] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:25.555446] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:25.561066] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:25.561110] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:25.585214] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:25.599197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.607174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.609268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.620089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.630560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.646350] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.648808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.653348] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.661246] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.661702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.671245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.671366] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.703665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.703702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.713077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:25.713783] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.733164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.756746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.774991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.809151] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:25.809220] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:25.810550] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:25.810609] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667462, retry interval=1000000, ret=-4179) [2022-08-08 11:04:25.814479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:25.823228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B5C-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:25.823436] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B5C-0-0] [lt=172] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:25.842890] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:25.842927] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:25.844781] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:25.844853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:25.845031] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:25.845289] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:25.845345] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667457, retry interval=1000000, ret=-4179) [2022-08-08 11:04:25.857821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.906361] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:25.916820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:25.989424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:26.009441] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:26.066802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:26.122253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:26.175587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:26.176702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:26.431269] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:26.563463] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:26.566687] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:26.566730] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:26.874172] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:26.874236] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667461, retry interval=1000000, ret=-4179) [2022-08-08 11:04:26.876797] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:26.876835] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667456, retry interval=1000000, ret=-4179) [2022-08-08 11:04:26.998273] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3421-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:26.998312] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3421-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:26.998749] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:26.998867] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:26.999010] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:26.999108] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:27.569322] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:27.572641] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:27.572667] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:27.816870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:27.876479] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CDE-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:27.876500] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CDE-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:27.903302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:27.904990] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:27.905053] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667460, retry interval=1000000, ret=-4179) [2022-08-08 11:04:27.906999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:27.907228] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:27.908746] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:27.908805] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667455, retry interval=1000000, ret=-4179) [2022-08-08 11:04:28.010068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3424-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:28.010155] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3424-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:28.575326] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:28.581714] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:28.581753] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:28.917349] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15339, time_dist: do_cluster_heartbeat=11, generate_cluster_version=15323) [2022-08-08 11:04:28.931716] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12690, time_dist: generate_cluster_version=12685) [2022-08-08 11:04:28.973489] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:28.973551] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667459, retry interval=1000000, ret=-4179) [2022-08-08 11:04:28.979985] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:28.980018] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667454, retry interval=1000000, ret=-4179) [2022-08-08 11:04:29.018989] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3427-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:29.019498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3427-0-0] [lt=477] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:29.019848] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:29.589783] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:29.593046] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:29.593084] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:29.703615] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10199) [2022-08-08 11:04:29.703897] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11174, time_dist: generate_cluster_version=11172) [2022-08-08 11:04:29.822239] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:04:29.822256] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:04:29.969229] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10528, time_dist: generate_cluster_version=10524) [2022-08-08 11:04:30.031518] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:30.031554] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667453, retry interval=1000000, ret=-4179) [2022-08-08 11:04:30.059688] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:30.059785] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667458, retry interval=1000000, ret=-4179) [2022-08-08 11:04:30.124635] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=244] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12397, time_dist: generate_cluster_version=12394) [2022-08-08 11:04:30.124675] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15003424}) [2022-08-08 11:04:30.343067] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=284] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927868367910, end_time=1659927870343054, elapsed_time=1975144, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:04:30.594932] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:30.598277] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:30.598315] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:30.814477] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:31.061199] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:31.061309] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:31.061830] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BA4-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:31.061953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BA4-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:31.062977] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:31.063032] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667452, retry interval=1000000, ret=-4179) [2022-08-08 11:04:31.075904] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12139) [2022-08-08 11:04:31.085585] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:31.085738] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:31.088026] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:31.088073] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667457, retry interval=1000000, ret=-4179) [2022-08-08 11:04:31.355517] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:31.384771] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:31.532649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=836] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:31.600985] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:31.605432] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:31.605478] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:31.649235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:31.690852] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:31.762963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:31.784995] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:31.795592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:31.798248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:31.807064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:31.817232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:31.916070] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=214] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:31.917014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:31.927318] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:32.049545] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927872049540, cur_ts_=1659927872029447, delta=20093) [2022-08-08 11:04:32.051563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:32.065677] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.066150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=451] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.066229] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.066994] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=296] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.067091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.067128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.067137] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.067161] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.067806] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.067910] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.068168] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B65-0-0] [lt=233] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:32.078098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:32.135997] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:32.136043] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667456, retry interval=1000000, ret=-4179) [2022-08-08 11:04:32.145767] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=2969] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:32.145833] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667451, retry interval=1000000, ret=-4179) [2022-08-08 11:04:32.164806] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:32.226991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:32.252913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:32.608791] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:32.612106] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:32.612179] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:32.691597] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:32.992169] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927872992162, cur_ts_=1659927872962292, delta=29870) [2022-08-08 11:04:33.047997] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3433-0-0] [lt=53] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:33.048180] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3433-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:33.147920] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BA7-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:33.151935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BA7-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:33.151964] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BA7-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:33.152003] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BA7-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:33.188837] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:33.188895] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667450, retry interval=1000000, ret=-4179) [2022-08-08 11:04:33.189668] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:33.189697] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667455, retry interval=1000000, ret=-4179) [2022-08-08 11:04:33.451103] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927873451093, cur_ts_=1659927873423247, delta=27846) [2022-08-08 11:04:33.614625] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:33.618493] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=460] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:33.618557] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:33.907738] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927873907732, cur_ts_=1659927873886562, delta=21170) [2022-08-08 11:04:34.223588] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927874223583, cur_ts_=1659927874194160, delta=29423) [2022-08-08 11:04:34.260729] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:34.261252] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:34.262538] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:34.262908] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=241] [dc=0] task process failed, start retry(max retry time=9223372036854667449, retry interval=1000000, ret=-4179) [2022-08-08 11:04:34.265030] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:34.265064] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667454, retry interval=1000000, ret=-4179) [2022-08-08 11:04:34.623625] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=385] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:34.627963] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:34.627999] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:35.016360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:35.060985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3439-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:35.140317] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.176532] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.189920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.293897] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:35.293953] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667448, retry interval=1000000, ret=-4179) [2022-08-08 11:04:35.300066] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:35.300104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667453, retry interval=1000000, ret=-4179) [2022-08-08 11:04:35.342917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:35.350123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.360292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.425585] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.425677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.446406] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:35.484294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:35.494864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.504956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:35.515498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.538633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.543170] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:35.547083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.553482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.563912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.567891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:35.568503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:35.569933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.608014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:35.620844] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.629839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.629967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.630924] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:35.634052] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:35.634094] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:35.640306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.645833] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.669761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.670942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.671668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.680279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.680439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.688398] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.694374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.722044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.723247] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.732077] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:35.733698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.756469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.773988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.791920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=135] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.830863] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927875830855, cur_ts_=1659927875795315, delta=35540) [2022-08-08 11:04:35.838281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:35.861563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:35.862149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:35.869522] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:35.882386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:35.925200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:35.934606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:36.008538] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:36.030445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:36.083455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:36.143236] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:36.194999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:36.195351] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:36.319202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:36.320703] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:36.320825] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667447, retry interval=1000000, ret=-4179) [2022-08-08 11:04:36.321904] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:36.323446] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:36.323485] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667452, retry interval=1000000, ret=-4179) [2022-08-08 11:04:36.362208] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:04:36.455738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:36.636535] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:36.640556] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:36.640616] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:36.778939] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927876778932, cur_ts_=1659927876748992, delta=29940) [2022-08-08 11:04:37.375006] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18760, time_dist: generate_cluster_version=18758) [2022-08-08 11:04:37.376587] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:37.376660] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667451, retry interval=1000000, ret=-4179) [2022-08-08 11:04:37.380064] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:37.380089] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:37.382102] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:37.382161] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667446, retry interval=1000000, ret=-4179) [2022-08-08 11:04:37.643221] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:37.646903] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:37.646947] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:37.814040] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15594, time_dist: generate_cluster_version=15591) [2022-08-08 11:04:37.840014] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:38.075504] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3442-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:38.075581] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3442-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:38.379220] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B6E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:38.379263] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B6E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:38.379286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B6E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:38.379335] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B6E-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:38.404066] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:38.404133] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667450, retry interval=1000000, ret=-4179) [2022-08-08 11:04:38.419781] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:38.419864] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667445, retry interval=1000000, ret=-4179) [2022-08-08 11:04:38.420140] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15236, time_dist: generate_cluster_version=15233) [2022-08-08 11:04:38.651754] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:38.655122] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:38.655148] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:38.741954] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11269) [2022-08-08 11:04:38.742170] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12203, time_dist: do_cluster_heartbeat=10, generate_cluster_version=12190) [2022-08-08 11:04:38.764860] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10822) [2022-08-08 11:04:38.765316] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12012, time_dist: do_cluster_heartbeat=8, generate_cluster_version=12001) [2022-08-08 11:04:38.942970] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12761, time_dist: generate_cluster_version=12756) [2022-08-08 11:04:38.967248] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17003) [2022-08-08 11:04:38.967430] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18005, time_dist: do_cluster_heartbeat=9, generate_cluster_version=17994) [2022-08-08 11:04:39.042949] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927879042940, cur_ts_=1659927879012617, delta=30323) [2022-08-08 11:04:39.082489] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3445-0-0] [lt=49] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:39.083009] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3445-0-0] [lt=501] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:39.168134] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16896) [2022-08-08 11:04:39.169271] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18876, time_dist: generate_cluster_version=18873) [2022-08-08 11:04:39.187800] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19620) [2022-08-08 11:04:39.188008] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=30979, time_dist: generate_cluster_version=30976) [2022-08-08 11:04:39.339153] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12386) [2022-08-08 11:04:39.339650] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14028, time_dist: do_cluster_heartbeat=9, generate_cluster_version=14015) [2022-08-08 11:04:39.410347] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=206] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13275) [2022-08-08 11:04:39.410635] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14544, time_dist: generate_cluster_version=14542) [2022-08-08 11:04:39.428110] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:39.432222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BB0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:39.432260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BB0-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:39.433132] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:39.434540] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] task process failed, start retry(max retry time=9223372036854667449, retry interval=1000000, ret=-4179) [2022-08-08 11:04:39.445424] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:39.445998] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:39.446029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667444, retry interval=1000000, ret=-4179) [2022-08-08 11:04:39.453628] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19511) [2022-08-08 11:04:39.474959] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16865) [2022-08-08 11:04:39.562775] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12210) [2022-08-08 11:04:39.562986] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13454, time_dist: generate_cluster_version=13453) [2022-08-08 11:04:39.658110] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:39.661232] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:39.661415] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:39.823361] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:04:39.823395] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:04:39.870405] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12232) [2022-08-08 11:04:40.047312] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15007712}) [2022-08-08 11:04:40.209103] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927880209095, cur_ts_=1659927880181376, delta=27719) [2022-08-08 11:04:40.210033] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39303) [2022-08-08 11:04:40.212562] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=43034, time_dist: generate_cluster_version=43030) [2022-08-08 11:04:40.292518] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927878368114, end_time=1659927880292257, elapsed_time=1924143, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:04:40.450670] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:40.450997] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF0-0-0] [lt=281] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:40.451084] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF0-0-0] [lt=58] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:40.468588] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:40.469415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:40.470587] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:40.470640] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667448, retry interval=1000000, ret=-4179) [2022-08-08 11:04:40.471443] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:40.471470] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667443, retry interval=1000000, ret=-4179) [2022-08-08 11:04:40.663871] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:40.669690] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:40.669759] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:40.874394] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:41.092814] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE344B-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:41.092911] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE344B-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:41.375993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:41.401176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:41.514329] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:41.514378] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:41.516317] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:41.516491] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] task process failed, start retry(max retry time=9223372036854667442, retry interval=1000000, ret=-4179) [2022-08-08 11:04:41.518430] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=128] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:41.518449] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:41.519744] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:41.519802] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667447, retry interval=1000000, ret=-4179) [2022-08-08 11:04:41.554281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:41.667296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:41.672265] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:41.675409] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:41.675452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:41.716130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:41.788427] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:41.808963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:41.819614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:41.822887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:41.830068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:41.840658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:41.936339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:41.936657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:41.946798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:42.084329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:42.096468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:42.179970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:42.242707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:42.277944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:42.548367] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11961, time_dist: generate_cluster_version=11959) [2022-08-08 11:04:42.592996] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:42.593038] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667441, retry interval=1000000, ret=-4179) [2022-08-08 11:04:42.598088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:42.598343] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=224] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:42.600431] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:42.600510] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667446, retry interval=1000000, ret=-4179) [2022-08-08 11:04:42.677666] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:42.682939] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:42.682985] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:42.864488] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1972] [dc=0] clock out of order(cur_ts=1659927882864476, cur_ts_=1659927882836321, delta=28155) [2022-08-08 11:04:43.130788] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927883130781, cur_ts_=1659927883098222, delta=32559) [2022-08-08 11:04:43.131910] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3451-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:43.131937] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3451-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:43.173403] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27077, time_dist: generate_cluster_version=27072) [2022-08-08 11:04:43.598554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:43.598630] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:43.610414] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6381-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:43.610448] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6381-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:43.644567] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927883644561, cur_ts_=1659927883616748, delta=27813) [2022-08-08 11:04:43.658816] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=1530] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:43.659085] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=121] [dc=0] task process failed, start retry(max retry time=9223372036854667445, retry interval=1000000, ret=-4179) [2022-08-08 11:04:43.659989] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=146] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:43.660247] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=225] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:43.662716] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:43.662808] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667440, retry interval=1000000, ret=-4179) [2022-08-08 11:04:43.689530] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:43.693137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:43.693170] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:44.365927] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927884365918, cur_ts_=1659927884343747, delta=22171) [2022-08-08 11:04:44.373587] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30496) [2022-08-08 11:04:44.373820] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32560, time_dist: generate_cluster_version=32557) [2022-08-08 11:04:44.387472] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13840) [2022-08-08 11:04:44.387690] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26504, time_dist: generate_cluster_version=26501) [2022-08-08 11:04:44.661512] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B77-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.661978] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B77-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.662123] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B77-0-0] [lt=105] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.662145] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B77-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.665170] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF6-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.666394] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF6-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.666419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF6-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.666519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF6-0-0] [lt=87] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.666674] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4CF6-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.690201] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.690221] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.690324] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:44.691673] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:44.691728] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667439, retry interval=1000000, ret=-4179) [2022-08-08 11:04:44.693402] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:44.693461] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667444, retry interval=1000000, ret=-4179) [2022-08-08 11:04:44.695916] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:44.700859] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:44.700911] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:45.038086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:45.153837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3457-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:45.153915] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3457-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:45.156329] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.197860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.213797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.363470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:45.375051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.375951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.443317] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.447349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.470944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:45.505457] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:45.512824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.526280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:45.538041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.561869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.568920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.569583] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:45.569922] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.583239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.589887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:45.590053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:45.590343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.632402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:45.644210] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.651745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.653225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.662609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.664096] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.688381] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.698653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.698898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.698932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.703797] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.704335] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:45.708999] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.710705] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:45.710975] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=255] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:45.717932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.724903] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:45.724936] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667443, retry interval=1000000, ret=-4179) [2022-08-08 11:04:45.725618] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:45.725640] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667438, retry interval=1000000, ret=-4179) [2022-08-08 11:04:45.738482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.738598] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.749966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.754309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:45.785141] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.795548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.807677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.857056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:45.877944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:45.878842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:45.893402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:45.898810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:45.941425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:45.954814] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:46.027847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:46.048543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:46.100629] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:46.165080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:46.214883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:46.214981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:46.478505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:46.713934] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:46.717681] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:46.717840] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=144] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:46.731120] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B7A-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:46.731546] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B7A-0-0] [lt=293] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:46.761661] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:46.761737] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667442, retry interval=1000000, ret=-4179) [2022-08-08 11:04:46.764286] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:46.765999] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:46.766031] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667437, retry interval=1000000, ret=-4179) [2022-08-08 11:04:47.163566] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE345D-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:47.163698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE345D-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:47.720345] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:47.723709] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:47.723813] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:47.817782] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:47.817810] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:47.819527] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:47.819612] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667441, retry interval=1000000, ret=-4179) [2022-08-08 11:04:47.821796] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10429, time_dist: generate_cluster_version=10426) [2022-08-08 11:04:47.827305] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:47.827367] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667436, retry interval=1000000, ret=-4179) [2022-08-08 11:04:47.865867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:48.272318] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2599] [dc=0] clock out of order(cur_ts=1659927888272313, cur_ts_=1659927888241123, delta=31190) [2022-08-08 11:04:48.726277] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:48.731735] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:48.731774] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:48.853473] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:48.853531] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667440, retry interval=1000000, ret=-4179) [2022-08-08 11:04:48.857011] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:48.857205] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:48.859067] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:48.859171] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667435, retry interval=1000000, ret=-4179) [2022-08-08 11:04:49.172809] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3463-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:49.734272] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:49.737572] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:49.737599] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:49.823045] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:04:49.823078] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:04:49.902969] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=106] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:49.903102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:49.903456] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=198] [dc=0] task process failed, start retry(max retry time=9223372036854667434, retry interval=1000000, ret=-4179) [2022-08-08 11:04:49.903259] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:49.904860] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:49.904906] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667439, retry interval=1000000, ret=-4179) [2022-08-08 11:04:49.972214] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15012096}) [2022-08-08 11:04:50.157109] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927888370329, end_time=1659927890157098, elapsed_time=1786769, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:04:50.179400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:50.739004] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:50.742370] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:50.742429] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:50.941494] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:50.946350] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:50.946386] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667433, retry interval=1000000, ret=-4179) [2022-08-08 11:04:50.955849] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:50.955910] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667438, retry interval=1000000, ret=-4179) [2022-08-08 11:04:51.185935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3469-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:51.401355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:51.421268] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:51.577900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:51.623561] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2523] [dc=0] clock out of order(cur_ts=1659927891623554, cur_ts_=1659927891601252, delta=22302) [2022-08-08 11:04:51.685537] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:51.740062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:51.748065] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:51.751655] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:51.751713] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:51.809940] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:51.830704] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:51.840812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:51.841319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:51.853536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:51.864422] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:51.952977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:51.956442] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:51.965604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:51.978660] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=216] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:51.978724] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667437, retry interval=1000000, ret=-4179) [2022-08-08 11:04:51.982796] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:51.984077] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:51.984177] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667432, retry interval=1000000, ret=-4179) [2022-08-08 11:04:52.109530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:52.129985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:52.191097] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE346C-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:52.191780] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:52.192134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:52.192316] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:52.195431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:52.267741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:52.298029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:52.691815] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:52.754869] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:52.758108] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:52.758148] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:53.031122] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:53.031156] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667431, retry interval=1000000, ret=-4179) [2022-08-08 11:04:53.034718] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:53.034817] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667436, retry interval=1000000, ret=-4179) [2022-08-08 11:04:53.216466] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] clock out of order(cur_ts=1659927893216459, cur_ts_=1659927893173420, delta=43039) [2022-08-08 11:04:53.760261] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:53.763328] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:53.763364] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:54.004593] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14050) [2022-08-08 11:04:54.052413] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12743) [2022-08-08 11:04:54.052824] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14856, time_dist: generate_cluster_version=14854) [2022-08-08 11:04:54.053779] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14969, time_dist: generate_cluster_version=14966) [2022-08-08 11:04:54.101202] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:54.101224] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:54.109267] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:54.110196] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] task process failed, start retry(max retry time=9223372036854667430, retry interval=1000000, ret=-4179) [2022-08-08 11:04:54.112838] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:54.112959] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667435, retry interval=1000000, ret=-4179) [2022-08-08 11:04:54.237201] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22571) [2022-08-08 11:04:54.237466] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23658, time_dist: do_cluster_heartbeat=10, generate_cluster_version=23646) [2022-08-08 11:04:54.282477] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11075, time_dist: generate_cluster_version=11070) [2022-08-08 11:04:54.303846] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] clock out of order(cur_ts=1659927894303840, cur_ts_=1659927894276603, delta=27237) [2022-08-08 11:04:54.582667] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18464) [2022-08-08 11:04:54.583098] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13892] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20658, time_dist: do_cluster_heartbeat=12, generate_cluster_version=20643) [2022-08-08 11:04:54.767640] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:54.773732] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:54.773758] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:55.056313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:55.162421] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927895162414, cur_ts_=1659927895130421, delta=31993) [2022-08-08 11:04:55.172780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.206812] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:55.206835] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:55.226371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.226643] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:55.226679] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667434, retry interval=1000000, ret=-4179) [2022-08-08 11:04:55.235829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.244190] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:55.244227] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667429, retry interval=1000000, ret=-4179) [2022-08-08 11:04:55.252748] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11627) [2022-08-08 11:04:55.253994] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659927895231090 2022-08-08 03:04:55.231090|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:13279845248159180005, inc:84118052, addr:{ip:"192.168.11.128", port:2882}, t:1659927895229000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F802A-0-0, start_time:1659927895230954, snapshot:1659927895226246|[end_task] u=9289 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=13 stc:1659927895240400|[calc_checksum_by_commit] u=22 ret:0, checksum:2501755618, checksum_log_ts:9223372036854775807|[submit_log] u=61 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=13433 trans_version:1659927895239961, uref:1073741825|[exiting] u=21 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:988659, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233102640|total_timeu=22864) [2022-08-08 11:04:55.261577] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13293, time_dist: generate_cluster_version=13290) [2022-08-08 11:04:55.261932] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11895, time_dist: generate_cluster_version=11893) [2022-08-08 11:04:55.383160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:55.392491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.394577] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.459106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.464156] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.490635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:55.521082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:55.531596] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.543518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:55.578738] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.579073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.589930] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.592081] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.592359] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] clock out of order(cur_ts=1659927895592355, cur_ts_=1659927895558349, delta=34006) [2022-08-08 11:04:55.590668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:55.599961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.610462] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=1136] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.611513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:04:55.612991] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:55.653983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:55.662980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.673415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.673580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.681435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.683669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.709614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.717093] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.717950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.719972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.722490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.728160] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.736341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.763057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.763283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.772017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.773041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:55.777944] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:55.781577] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:55.781627] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:55.810256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.819768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.830277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=133] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.892720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:55.894163] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:04:55.894757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:04:55.912875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:55.915894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:55.958164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:55.978780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:56.047945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:56.072201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:56.115977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:04:56.189591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:56.237692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:04:56.241023] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:56.268038] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659927896268032, cur_ts_=1659927896240809, delta=27223) [2022-08-08 11:04:56.285948] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=1326] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16589) [2022-08-08 11:04:56.286893] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19021, time_dist: generate_cluster_version=19018) [2022-08-08 11:04:56.293562] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=243] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:56.293723] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22905, time_dist: do_cluster_heartbeat=6, generate_cluster_version=22897) [2022-08-08 11:04:56.297784] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] task process failed, start retry(max retry time=9223372036854667433, retry interval=1000000, ret=-4179) [2022-08-08 11:04:56.303341] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:56.303380] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667428, retry interval=1000000, ret=-4179) [2022-08-08 11:04:56.309455] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10960) [2022-08-08 11:04:56.411584] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:04:56.502758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:04:56.784266] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:56.788006] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:56.788102] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:57.265014] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE347B-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:57.331251] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:57.332007] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667427, retry interval=1000000, ret=-4179) [2022-08-08 11:04:57.342766] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:57.343028] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:57.344598] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:57.344803] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=151] [dc=0] task process failed, start retry(max retry time=9223372036854667432, retry interval=1000000, ret=-4179) [2022-08-08 11:04:57.791231] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:57.796734] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:57.796760] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:57.881439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:04:58.009512] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15708) [2022-08-08 11:04:58.009791] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17670, time_dist: do_cluster_heartbeat=15, generate_cluster_version=17652) [2022-08-08 11:04:58.236217] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] [trans sampling] (TRACE=begin_ts=1659927898231465 2022-08-08 03:04:58.231465|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:7555518151974462156, inc:84118162, addr:{ip:"192.168.11.128", port:2882}, t:1659927898231000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC73A-0-0, start_time:1659927898231351, snapshot:1659927898146055|[end_task] u=171 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659927898231655|[calc_checksum_by_commit] u=20 ret:0, checksum:2714690280, checksum_log_ts:9223372036854775807|[submit_log] u=87 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=4345 trans_version:1659927898231181, uref:1073741825|[exiting] u=26 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=4 ret:0, log_type:64, log_id:8317899, logging:0, pending:0, uref:1|[end_trans_cb] u=25 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=4706) [2022-08-08 11:04:58.269578] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] clock out of order(cur_ts=1659927898269571, cur_ts_=1659927898242287, delta=27284) [2022-08-08 11:04:58.281157] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11272) [2022-08-08 11:04:58.291698] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22850, time_dist: do_cluster_heartbeat=12, generate_cluster_version=22833) [2022-08-08 11:04:58.452479] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927898452474, cur_ts_=1659927898422959, delta=29515) [2022-08-08 11:04:58.460852] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:58.461344] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] task process failed, start retry(max retry time=9223372036854667426, retry interval=1000000, ret=-4179) [2022-08-08 11:04:58.487753] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:58.487791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:58.521381] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=915] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:58.521437] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667431, retry interval=1000000, ret=-4179) [2022-08-08 11:04:58.531722] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=202] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=56481) [2022-08-08 11:04:58.554045] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22276) [2022-08-08 11:04:58.555154] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=82401, time_dist: generate_cluster_version=82397) [2022-08-08 11:04:58.565191] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=93298, time_dist: generate_cluster_version=93294) [2022-08-08 11:04:58.571632] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=176] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17367) [2022-08-08 11:04:58.651010] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=153] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14845) [2022-08-08 11:04:58.653375] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35697, time_dist: do_cluster_heartbeat=12, generate_cluster_version=35683) [2022-08-08 11:04:58.661640] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10356) [2022-08-08 11:04:58.661860] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=43013, time_dist: do_cluster_heartbeat=8, generate_cluster_version=42996, cluster_service_self_check=6) [2022-08-08 11:04:58.800483] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:58.805091] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:58.805130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:58.893138] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19786, time_dist: generate_cluster_version=19783) [2022-08-08 11:04:59.017219] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11524) [2022-08-08 11:04:59.017858] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12926, time_dist: do_cluster_heartbeat=11, generate_cluster_version=12911) [2022-08-08 11:04:59.037524] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=130] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20072) [2022-08-08 11:04:59.038738] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32751, time_dist: do_cluster_heartbeat=7, generate_cluster_version=32743) [2022-08-08 11:04:59.264743] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927899264737, cur_ts_=1659927899233968, delta=30769) [2022-08-08 11:04:59.283706] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=815] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18580) [2022-08-08 11:04:59.284039] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23057, time_dist: generate_cluster_version=23053) [2022-08-08 11:04:59.293069] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3481-0-0] [lt=94] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:59.293114] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3481-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:04:59.310170] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26395) [2022-08-08 11:04:59.311698] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=49831, time_dist: generate_cluster_version=49827) [2022-08-08 11:04:59.498249] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927899498243, cur_ts_=1659927899472660, delta=25583) [2022-08-08 11:04:59.532780] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:59.532972] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:59.536316] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:59.536549] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=162] [dc=0] task process failed, start retry(max retry time=9223372036854667425, retry interval=1000000, ret=-4179) [2022-08-08 11:04:59.539163] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D0D-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:59.539283] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D0D-0-0] [lt=72] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:59.580313] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:59.580438] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:04:59.581908] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:04:59.582532] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=470] [dc=0] task process failed, start retry(max retry time=9223372036854667430, retry interval=1000000, ret=-4179) [2022-08-08 11:04:59.598163] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15319) [2022-08-08 11:04:59.664305] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=5756] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11596) [2022-08-08 11:04:59.668715] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17592, time_dist: generate_cluster_version=17588) [2022-08-08 11:04:59.681931] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15637) [2022-08-08 11:04:59.682271] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20066, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20055) [2022-08-08 11:04:59.813990] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:04:59.832901] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:04:59.832944] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:04:59.846220] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:04:59.846243] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:04:59.859064] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16079) [2022-08-08 11:04:59.859332] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=1993] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17931, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17918) [2022-08-08 11:04:59.902160] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18511) [2022-08-08 11:04:59.902394] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19555, time_dist: generate_cluster_version=19552) [2022-08-08 11:05:00.037289] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] clock out of order(cur_ts=1659927900037280, cur_ts_=1659927900004859, delta=32421) [2022-08-08 11:05:00.068539] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11784) [2022-08-08 11:05:00.089835] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21035) [2022-08-08 11:05:00.090146] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28912, time_dist: do_cluster_heartbeat=10, generate_cluster_version=28899) [2022-08-08 11:05:00.107977] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17761) [2022-08-08 11:05:00.108311] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35554, time_dist: generate_cluster_version=35551) [2022-08-08 11:05:00.177997] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927900177988, cur_ts_=1659927900152769, delta=25219) [2022-08-08 11:05:00.226457] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12524) [2022-08-08 11:05:00.226765] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14456, time_dist: do_cluster_heartbeat=11, generate_cluster_version=14443) [2022-08-08 11:05:00.281884] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11945) [2022-08-08 11:05:00.282141] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25511, time_dist: generate_cluster_version=25508) [2022-08-08 11:05:00.312619] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927900312611, cur_ts_=1659927900287965, delta=24646) [2022-08-08 11:05:00.565211] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:00.565927] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667424, retry interval=1000000, ret=-4179) [2022-08-08 11:05:00.585212] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6399-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:00.585283] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6399-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:00.606341] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:00.606537] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667429, retry interval=1000000, ret=-4179) [2022-08-08 11:05:00.686054] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12001) [2022-08-08 11:05:00.686464] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14474, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14461) [2022-08-08 11:05:00.697805] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11676) [2022-08-08 11:05:00.698032] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25295, time_dist: generate_cluster_version=25293) [2022-08-08 11:05:00.760900] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927900760890, cur_ts_=1659927900731086, delta=29804) [2022-08-08 11:05:00.840088] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:00.844234] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:00.844259] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:00.867578] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15016480}) [2022-08-08 11:05:00.904001] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927900903993, cur_ts_=1659927900882240, delta=21753) [2022-08-08 11:05:00.942806] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10248) [2022-08-08 11:05:00.943010] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13133, time_dist: do_cluster_heartbeat=8, generate_cluster_version=13123) [2022-08-08 11:05:01.010995] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927901010990, cur_ts_=1659927900988787, delta=22203) [2022-08-08 11:05:01.034348] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:01.050414] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15097) [2022-08-08 11:05:01.093415] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927898393821, end_time=1659927901093403, elapsed_time=2699582, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:05:01.279494] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=23] [dc=0] [trans sampling] (TRACE=begin_ts=1659927901278653 2022-08-08 03:05:01.278653|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:7549184569317210791, inc:84118284, addr:{ip:"192.168.11.128", port:2882}, t:1659927901277000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6976-0-0, start_time:1659927901278532, snapshot:1659927901193267|[end_task] u=125 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927901278796|[calc_checksum_by_commit] u=21 ret:0, checksum:1382280861, checksum_log_ts:9223372036854775807|[submit_log] u=66 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=487 trans_version:1659927901278817, uref:1073741825|[exiting] u=19 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8303316, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232375600|total_timeu=762) [2022-08-08 11:05:01.323068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3487-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:01.323185] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3487-0-0] [lt=62] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:01.421535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:01.442231] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:01.568231] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D10-0-0] [lt=243] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.568420] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D10-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.568542] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D10-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.594936] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.595047] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.596646] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:01.596693] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667423, retry interval=1000000, ret=-4179) [2022-08-08 11:05:01.596972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:01.608756] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B8F-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.608959] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4B8F-0-0] [lt=182] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:01.637598] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:01.637640] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667428, retry interval=1000000, ret=-4179) [2022-08-08 11:05:01.700735] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=432] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:01.756685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:01.828080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:01.847033] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:01.851699] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:01.852867] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:01.852892] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:01.859821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:01.860374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:01.871253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:01.889417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:01.974383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:01.975990] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:01.986371] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:02.133327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:02.151165] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:02.219621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:02.287378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:02.322286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:02.331267] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE348A-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:02.331836] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:02.622968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:02.623152] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:02.624454] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:02.624538] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667422, retry interval=1000000, ret=-4179) [2022-08-08 11:05:02.659489] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:02.659724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:02.661092] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:02.661167] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667427, retry interval=1000000, ret=-4179) [2022-08-08 11:05:02.857271] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:02.860357] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:02.860393] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:03.339093] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE348D-0-0] [lt=183] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:03.339539] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE348D-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:03.675343] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:03.675436] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667421, retry interval=1000000, ret=-4179) [2022-08-08 11:05:03.717128] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927903717118, cur_ts_=1659927903685822, delta=31296) [2022-08-08 11:05:03.724247] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:03.724310] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667426, retry interval=1000000, ret=-4179) [2022-08-08 11:05:03.724529] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=49459) [2022-08-08 11:05:03.726017] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=58639, time_dist: generate_cluster_version=58635) [2022-08-08 11:05:03.862725] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:03.866653] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:03.873689] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=6977] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:04.704138] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:04.704199] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667420, retry interval=1000000, ret=-4179) [2022-08-08 11:05:04.750952] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927904750931, cur_ts_=1659927904723225, delta=27706) [2022-08-08 11:05:04.766442] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:04.766522] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] task process failed, start retry(max retry time=9223372036854667425, retry interval=1000000, ret=-4179) [2022-08-08 11:05:04.877534] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:04.881146] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:04.881185] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:05.073018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:05.197926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.249275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.259823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.347617] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3493-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:05.405267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:05.412707] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.416048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.480660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.488581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.510379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=196] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:05.539308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:05.554065] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.560203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:05.601288] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.602054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.612667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=2404] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.614576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=7173] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:05.614591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.621873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.632548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:05.633502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.633568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:05.674876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:05.685319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.695448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.697551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=134] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.698959] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.707047] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D16-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.707134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D16-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.707291] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D16-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.707688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.708117] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D16-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.726396] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.728739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.730172] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:05.730214] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667419, retry interval=1000000, ret=-4179) [2022-08-08 11:05:05.736572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.736633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.737113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.747076] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.750487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.757455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.781896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.790685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.790774] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.790789] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:05.792202] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.792871] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:05.794314] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:05.794379] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667424, retry interval=1000000, ret=-4179) [2022-08-08 11:05:05.832421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.842934] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.847958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.883900] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:05.888954] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:05.888999] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:05.915095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=412] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:05.915227] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:05.919296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:05.935049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:05.941432] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:05.976804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:05.996232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:06.070416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:06.093271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:06.134021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:06.210898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:06.259579] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:06.259764] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:06.520416] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:06.732621] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BD7-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:06.732655] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BD7-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:06.732712] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BD7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:06.732773] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BD7-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:06.732923] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BD7-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:06.777761] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:06.777821] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667418, retry interval=1000000, ret=-4179) [2022-08-08 11:05:06.814708] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:06.814774] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667423, retry interval=1000000, ret=-4179) [2022-08-08 11:05:06.892738] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:06.896130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:06.896222] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:07.358883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3499-0-0] [lt=64] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:07.359576] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:07.359822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:07.359857] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:07.783497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:07.801894] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:07.801933] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:07.804273] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:07.804402] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] task process failed, start retry(max retry time=9223372036854667417, retry interval=1000000, ret=-4179) [2022-08-08 11:05:07.816450] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D19-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:07.816623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D19-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:07.834486] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:07.835996] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:07.836049] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667422, retry interval=1000000, ret=-4179) [2022-08-08 11:05:07.904785] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:07.905449] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:07.914054] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27315) [2022-08-08 11:05:07.914420] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=35488, time_dist: do_cluster_heartbeat=11, generate_cluster_version=35475) [2022-08-08 11:05:07.920454] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:07.920479] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:07.931956] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17861) [2022-08-08 11:05:07.933177] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=42841, time_dist: generate_cluster_version=42837) [2022-08-08 11:05:08.238483] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:05:08.848758] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:08.848894] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667416, retry interval=1000000, ret=-4179) [2022-08-08 11:05:08.865033] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:08.865092] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23191) [2022-08-08 11:05:08.865240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:08.869322] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:08.869433] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667421, retry interval=1000000, ret=-4179) [2022-08-08 11:05:08.870391] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=29092, time_dist: do_cluster_heartbeat=4, generate_cluster_version=29084) [2022-08-08 11:05:08.916606] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2389] [dc=0] clock out of order(cur_ts=1659927908916589, cur_ts_=1659927908890660, delta=25929) [2022-08-08 11:05:08.926647] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:08.928243] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=33984) [2022-08-08 11:05:08.928311] WARN [STORAGE.TRANS] submit_log_impl_ (ob_trans_part_ctx.cpp:6697) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] submitting log costs too much time(ret=0, ret="OB_SUCCESS", timeguard=time guard 'submit_log_impl' cost too much time, used=34052, time_dist: 2,20,34026,2, trans_id={hash:17863386638012208245, inc:84118577, addr:"192.168.11.128:2882", t:1659927908890660}, partition={tid:1099511628002, partition_id:0, part_cnt:0}, state={prepare_version:1659927908890660, state:0}, real_log_type=64, log_size=464) [2022-08-08 11:05:08.928323] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard 'submit_log_impl' cost too much time, used=34102, time_dist: 2,20,34026,2) [2022-08-08 11:05:08.934855] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:08.934894] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:08.939165] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10640) [2022-08-08 11:05:08.966943] TRACE [TRACE]ob_trans_ctx.cpp:487 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=22] [dc=0] [trans warn] (TRACE=begin_ts=1659927908894020 2022-08-08 03:05:08.894020|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:17863386638012208245, inc:84118577, addr:{ip:"192.168.11.128", port:2882}, t:1659927908890660}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8AC76F-0-0, start_time:1659927908893916, snapshot:1659927908871312|[end_task] u=179 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659927908894215|[calc_checksum_by_commit] u=18 ret:0, checksum:1886153688, checksum_log_ts:9223372036854775807|[submit_log] u=34032 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=61 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=38517 trans_version:1659927908890660, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8317952, logging:0, pending:0, uref:1|[end_trans_cb] u=9 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=72858) [2022-08-08 11:05:08.969211] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=75613, time_dist: generate_cluster_version=75610) [2022-08-08 11:05:09.367142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE349F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:09.367268] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE349F-0-0] [lt=69] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:09.435256] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927909435250, cur_ts_=1659927909410970, delta=24280) [2022-08-08 11:05:09.630275] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927909630270, cur_ts_=1659927909603989, delta=26281) [2022-08-08 11:05:09.823939] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:05:09.823956] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:05:09.892749] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:09.893317] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667420, retry interval=1000000, ret=-4179) [2022-08-08 11:05:09.902241] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:09.902275] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667415, retry interval=1000000, ret=-4179) [2022-08-08 11:05:09.937708] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:09.940696] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:09.940734] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:10.106894] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11070, time_dist: generate_cluster_version=11068) [2022-08-08 11:05:10.192039] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927910192030, cur_ts_=1659927910158487, delta=33543) [2022-08-08 11:05:10.210693] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15020864}) [2022-08-08 11:05:10.405995] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927908394166, end_time=1659927910405984, elapsed_time=2011818, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:05:10.905222] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BDD-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:10.942931] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:10.955111] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:10.955146] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667419, retry interval=1000000, ret=-4179) [2022-08-08 11:05:10.955603] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:10.955620] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:10.972453] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:10.972505] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667414, retry interval=1000000, ret=-4179) [2022-08-08 11:05:11.116571] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:11.436777] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:11.462949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:11.612304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:11.717528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:11.776672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:11.853240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:11.869920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:11.875685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:11.880168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:11.890697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:11.911754] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:11.957721] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D1F-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:11.958522] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:11.963940] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:11.964016] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:11.991969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:11.993284] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:11.993515] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667418, retry interval=1000000, ret=-4179) [2022-08-08 11:05:11.999281] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:11.999346] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667413, retry interval=1000000, ret=-4179) [2022-08-08 11:05:11.999519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:12.007235] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:12.152965] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:12.176514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:12.241848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:12.302611] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:12.344486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:12.386913] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34A8-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:12.387068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34A8-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:12.692208] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:12.858254] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659927912858247, cur_ts_=1659927912835037, delta=23210) [2022-08-08 11:05:12.967443] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:12.970620] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:12.970665] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:13.000636] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:13.000863] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:13.001807] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BE0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:13.028829] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:13.028862] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667417, retry interval=1000000, ret=-4179) [2022-08-08 11:05:13.032534] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:13.032570] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667412, retry interval=1000000, ret=-4179) [2022-08-08 11:05:13.394855] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34AB-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:13.395567] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34AB-0-0] [lt=693] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:13.396620] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:13.396643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:13.396656] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:13.972898] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:13.976104] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:13.976152] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:14.069300] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:14.069336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:14.070804] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:14.070839] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667411, retry interval=1000000, ret=-4179) [2022-08-08 11:05:14.073791] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:14.074075] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:14.075286] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:14.075332] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667416, retry interval=1000000, ret=-4179) [2022-08-08 11:05:14.216181] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] clock out of order(cur_ts=1659927914216176, cur_ts_=1659927914193377, delta=22799) [2022-08-08 11:05:14.335651] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927914335646, cur_ts_=1659927914306731, delta=28915) [2022-08-08 11:05:14.351042] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16073, time_dist: generate_cluster_version=16069) [2022-08-08 11:05:14.980687] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:14.983965] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:14.984002] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:15.091338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=317] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:15.094961] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:15.095159] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:15.096235] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:15.096270] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667415, retry interval=1000000, ret=-4179) [2022-08-08 11:05:15.098313] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:15.098351] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667410, retry interval=1000000, ret=-4179) [2022-08-08 11:05:15.122778] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20917, time_dist: generate_cluster_version=20913) [2022-08-08 11:05:15.217808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.264911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.279927] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.408592] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34B1-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:15.408726] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34B1-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:15.430708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:15.432443] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.440006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=347] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.502568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.505062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.534918] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:15.559790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:15.569460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.580420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:15.621558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=158] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.622491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.632456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:15.637479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.639433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.639934] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.650304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:15.652792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:15.653518] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.691461] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:15.704967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.715044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.715292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.723739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.733172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.749494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.756460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.757727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.758033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.762599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.766012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.774708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.799292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.809567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.809651] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.814502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:15.851487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.866594] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.872095] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.932056] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:15.934304] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:15.934530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:15.956490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:15.965352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:15.986338] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:15.989936] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:15.990077] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:15.992125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:16.011601] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:16.090765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:16.112646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:16.126786] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:16.126819] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667414, retry interval=1000000, ret=-4179) [2022-08-08 11:05:16.131239] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:16.131298] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667409, retry interval=1000000, ret=-4179) [2022-08-08 11:05:16.155560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:16.233148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:16.281772] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:16.284710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:16.443971] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:05:16.543348] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:16.743732] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927916743724, cur_ts_=1659927916721198, delta=22526) [2022-08-08 11:05:16.992999] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:16.997458] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:16.997497] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:17.133832] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63B1-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:17.153157] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:17.153535] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667413, retry interval=1000000, ret=-4179) [2022-08-08 11:05:17.156533] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:17.157867] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:17.157940] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667408, retry interval=1000000, ret=-4179) [2022-08-08 11:05:17.420100] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34B7-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:17.420135] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34B7-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:17.922623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:18.000351] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:18.003337] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:18.003372] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:18.159554] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D28-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:18.182776] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:18.183869] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:18.183929] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667407, retry interval=1000000, ret=-4179) [2022-08-08 11:05:18.185294] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:18.185359] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667412, retry interval=1000000, ret=-4179) [2022-08-08 11:05:19.005802] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:19.009622] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:19.009797] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:19.210137] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:19.210174] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667411, retry interval=1000000, ret=-4179) [2022-08-08 11:05:19.225019] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:19.225053] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667406, retry interval=1000000, ret=-4179) [2022-08-08 11:05:19.232905] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27481] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21466) [2022-08-08 11:05:19.465218] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927919465205, cur_ts_=1659927919443300, delta=21905) [2022-08-08 11:05:19.824529] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:05:19.824560] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:05:20.011674] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:20.015200] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:20.015253] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:20.031869] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15025248}) [2022-08-08 11:05:20.214486] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927918395652, end_time=1659927920214476, elapsed_time=1818824, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:05:20.227571] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4BAA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:20.227803] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4BAA-0-0] [lt=206] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:20.234407] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:20.234548] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:20.235723] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:20.235769] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667410, retry interval=1000000, ret=-4179) [2022-08-08 11:05:20.243907] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:20.244159] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:20.251058] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:20.251125] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667405, retry interval=1000000, ret=-4179) [2022-08-08 11:05:21.019690] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:21.022734] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:21.022880] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:21.196096] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:21.237902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63B7-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:21.238067] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63B7-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:21.262142] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=933] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:21.262207] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667409, retry interval=1000000, ret=-4179) [2022-08-08 11:05:21.271112] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:21.272509] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:21.272568] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667404, retry interval=1000000, ret=-4179) [2022-08-08 11:05:21.457086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:21.488321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:21.635093] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:21.738761] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:21.801693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:21.874136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:21.890368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:21.896955] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:21.899197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:21.907357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:21.928300] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:22.011543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:22.022197] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:22.027434] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:22.029407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:22.030400] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:22.030433] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:22.177277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:22.197253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:22.260850] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:22.264361] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D2E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:22.264575] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D2E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:22.285410] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=172] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:22.285468] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667408, retry interval=1000000, ret=-4179) [2022-08-08 11:05:22.300134] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:22.301912] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:22.301976] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667403, retry interval=1000000, ret=-4179) [2022-08-08 11:05:22.322039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:22.363294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:23.032798] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:23.036145] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:23.036188] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:23.303482] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:23.303715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:23.303909] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BF1-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:23.307755] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:23.307810] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667407, retry interval=1000000, ret=-4179) [2022-08-08 11:05:23.322476] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:23.322542] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667402, retry interval=1000000, ret=-4179) [2022-08-08 11:05:24.038790] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:24.041872] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:24.041910] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:24.283906] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] clock out of order(cur_ts=1659927924283900, cur_ts_=1659927924260731, delta=23169) [2022-08-08 11:05:24.311387] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D31-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.311446] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D31-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.325710] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4BB0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.326052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4BB0-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.329722] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.329759] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.331384] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:24.331446] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667406, retry interval=1000000, ret=-4179) [2022-08-08 11:05:24.345166] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12678) [2022-08-08 11:05:24.348258] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.348304] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:24.350145] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:24.350452] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=258] [dc=0] task process failed, start retry(max retry time=9223372036854667401, retry interval=1000000, ret=-4179) [2022-08-08 11:05:24.683512] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927924683502, cur_ts_=1659927924662566, delta=20936) [2022-08-08 11:05:25.044471] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:25.048667] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:25.048722] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:25.108477] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:25.242821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.282513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.299060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.333302] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63BD-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:25.333425] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63BD-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:25.354961] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=320] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:25.355003] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667405, retry interval=1000000, ret=-4179) [2022-08-08 11:05:25.354868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BF4-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:25.355167] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BF4-0-0] [lt=218] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:25.372109] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16410) [2022-08-08 11:05:25.372921] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:25.373141] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:25.374341] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:25.374403] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667400, retry interval=1000000, ret=-4179) [2022-08-08 11:05:25.451756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:25.456281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.463503] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.525907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.525944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.557913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:25.578089] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:25.590041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.598470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:25.641326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.645326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.655600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:25.655653] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.661949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.663950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.665992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:25.676360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:25.679569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.710116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:25.727086] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.730983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.738868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.746128] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.750690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.773746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.773673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.775582] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=182] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.781951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.782457] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.782486] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.794669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=490] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.824105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.834627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.834650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:25.834743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.876174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.889144] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.896730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.952558] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:25.953737] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:25.959085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:25.974925] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:25.982825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:26.015113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:26.036900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:26.051281] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:26.055152] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:26.055238] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:26.115243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:26.128702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:26.176966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:26.249334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:26.300271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:26.309366] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:26.375523] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:26.375722] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:26.376990] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:26.377051] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667404, retry interval=1000000, ret=-4179) [2022-08-08 11:05:26.394861] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:26.395018] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=107] [dc=0] task process failed, start retry(max retry time=9223372036854667399, retry interval=1000000, ret=-4179) [2022-08-08 11:05:26.493833] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1533] [dc=0] clock out of order(cur_ts=1659927926493823, cur_ts_=1659927926471451, delta=22372) [2022-08-08 11:05:26.567145] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:27.059056] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:27.062533] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:27.062559] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:27.397068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63C0-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:27.397301] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63C0-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:27.398223] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:27.398371] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:27.399569] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:27.399604] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667403, retry interval=1000000, ret=-4179) [2022-08-08 11:05:27.417333] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:27.417385] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667398, retry interval=1000000, ret=-4179) [2022-08-08 11:05:27.471529] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34D5-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:27.471606] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34D5-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:27.472208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:27.472233] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:27.472586] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:27.946661] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:28.065033] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:28.068226] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:28.068262] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:28.402984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D37-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:28.403162] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D37-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:28.417290] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:28.417326] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:28.418667] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:28.418700] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667402, retry interval=1000000, ret=-4179) [2022-08-08 11:05:28.441460] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:28.441515] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667397, retry interval=1000000, ret=-4179) [2022-08-08 11:05:29.022002] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927929021996, cur_ts_=1659927928996262, delta=25734) [2022-08-08 11:05:29.071083] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:29.074292] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:29.074317] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:29.420947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4BFA-0-0] [lt=48] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:29.450258] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927929450253, cur_ts_=1659927929424239, delta=26014) [2022-08-08 11:05:29.475453] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:29.475493] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667401, retry interval=1000000, ret=-4179) [2022-08-08 11:05:29.477597] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:29.477639] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667396, retry interval=1000000, ret=-4179) [2022-08-08 11:05:29.824996] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:05:29.825036] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:05:30.050779] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15029568}) [2022-08-08 11:05:30.081804] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927930081796, cur_ts_=1659927930049468, delta=32328) [2022-08-08 11:05:30.084660] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:30.090170] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:30.090219] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:30.266389] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927928395943, end_time=1659927930266373, elapsed_time=1870430, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:05:30.500940] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:30.501060] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667395, retry interval=1000000, ret=-4179) [2022-08-08 11:05:30.505840] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:30.505896] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667400, retry interval=1000000, ret=-4179) [2022-08-08 11:05:31.093025] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:31.096152] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:31.096217] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:31.257074] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:31.473722] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:31.510542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:31.549335] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:31.549375] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667399, retry interval=1000000, ret=-4179) [2022-08-08 11:05:31.550743] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:31.550896] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:31.552553] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:31.552597] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667394, retry interval=1000000, ret=-4179) [2022-08-08 11:05:31.652307] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:31.762049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:31.819421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:31.895829] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:31.908221] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:31.916739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:31.918623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:31.928066] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:31.948241] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:32.032802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:32.041614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:32.045814] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:32.099233] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:32.104700] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:32.104728] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:32.195644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:32.214880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:32.279190] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:32.342688] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:32.383642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:32.551160] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4BBC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.557507] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D3E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.557789] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D3E-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.572703] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.572951] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.576042] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:32.576105] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667398, retry interval=1000000, ret=-4179) [2022-08-08 11:05:32.576332] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.576452] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:32.577812] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:32.577874] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667393, retry interval=1000000, ret=-4179) [2022-08-08 11:05:32.692478] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:32.954172] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927932954162, cur_ts_=1659927932916027, delta=38135) [2022-08-08 11:05:33.109018] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:33.113063] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:33.113104] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:33.578420] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63C9-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:33.600040] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=338] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:33.607282] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:33.607398] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667392, retry interval=1000000, ret=-4179) [2022-08-08 11:05:33.609094] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:33.609168] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] task process failed, start retry(max retry time=9223372036854667397, retry interval=1000000, ret=-4179) [2022-08-08 11:05:33.716368] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=64] [dc=0] clock out of order(cur_ts=1659927933716360, cur_ts_=1659927933685851, delta=30509) [2022-08-08 11:05:34.115968] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:34.119610] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:34.119658] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:34.609795] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4BBF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:34.611296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D41-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:34.634689] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:34.636008] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:34.636054] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667396, retry interval=1000000, ret=-4179) [2022-08-08 11:05:34.637904] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:34.637951] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667391, retry interval=1000000, ret=-4179) [2022-08-08 11:05:35.136890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:35.137109] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:35.143608] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:35.143633] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:35.221635] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659927935221631, cur_ts_=1659927935195236, delta=26395) [2022-08-08 11:05:35.232193] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11195, time_dist: do_cluster_heartbeat=6, generate_cluster_version=11186) [2022-08-08 11:05:35.267243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.303723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.325650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.471467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:35.477566] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.487875] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.548390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.556343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.579677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:35.599540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:35.608551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=595] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.618897] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:35.656664] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:35.656698] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667395, retry interval=1000000, ret=-4179) [2022-08-08 11:05:35.661124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.667627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.672228] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:35.672261] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667390, retry interval=1000000, ret=-4179) [2022-08-08 11:05:35.677369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.677517] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:35.677910] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.686814] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:35.687855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.695311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:35.701958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.728608] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:35.750249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.752183] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.760825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.764203] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.773538] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.794052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=161] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.795481] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.798913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.804571] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.804733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.804895] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.811208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.840483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.849944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.850018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.856358] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:35.898233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.910944] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.918887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:35.971311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:35.977281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:35.981463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:35.995958] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.002232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:36.034569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:36.035919] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17759, time_dist: generate_cluster_version=17756) [2022-08-08 11:05:36.059278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.138142] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.150533] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:36.154332] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:36.154372] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:36.156655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.201369] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:36.266563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.324168] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:36.325109] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.469760] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:05:36.509772] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34F0-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:36.509793] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34F0-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:36.586807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:36.660460] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D44-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:36.684586] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:36.684980] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] task process failed, start retry(max retry time=9223372036854667394, retry interval=1000000, ret=-4179) [2022-08-08 11:05:36.695791] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=905] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:36.695948] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] task process failed, start retry(max retry time=9223372036854667389, retry interval=1000000, ret=-4179) [2022-08-08 11:05:37.156875] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:37.160014] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:37.160052] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:37.517573] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34F3-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:37.517729] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34F3-0-0] [lt=83] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:37.687129] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C06-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:37.698208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63CF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:37.707319] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:37.707367] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667393, retry interval=1000000, ret=-4179) [2022-08-08 11:05:37.718757] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:37.720348] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:37.720384] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667388, retry interval=1000000, ret=-4179) [2022-08-08 11:05:37.966323] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:38.163033] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:38.166690] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:38.166734] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:38.288564] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927938288559, cur_ts_=1659927938257760, delta=30799) [2022-08-08 11:05:38.714676] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D47-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:38.714732] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D47-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:38.714745] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D47-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:38.715058] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D47-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:38.758466] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=160] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:38.758559] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667387, retry interval=1000000, ret=-4179) [2022-08-08 11:05:38.760200] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:38.760228] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667392, retry interval=1000000, ret=-4179) [2022-08-08 11:05:39.168905] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:39.171985] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:39.172022] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:39.278362] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=157] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18622) [2022-08-08 11:05:39.278775] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24905, time_dist: generate_cluster_version=24901) [2022-08-08 11:05:39.802412] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:39.802462] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667386, retry interval=1000000, ret=-4179) [2022-08-08 11:05:39.804386] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:39.804417] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667391, retry interval=1000000, ret=-4179) [2022-08-08 11:05:39.826728] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:05:39.826778] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:05:39.882709] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927939882703, cur_ts_=1659927939852899, delta=29804) [2022-08-08 11:05:40.102214] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15033888}) [2022-08-08 11:05:40.173804] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:40.176906] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:40.176945] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:40.283750] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927938396987, end_time=1659927940283717, elapsed_time=1886730, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:05:40.374470] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927940374458, cur_ts_=1659927940353293, delta=21165) [2022-08-08 11:05:40.830304] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:40.830338] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667390, retry interval=1000000, ret=-4179) [2022-08-08 11:05:40.830846] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:40.832466] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:40.832499] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667385, retry interval=1000000, ret=-4179) [2022-08-08 11:05:41.180165] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:41.186127] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:41.186164] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:41.346636] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:41.495988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:41.530087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:41.549772] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34FF-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.549806] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34FF-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.550519] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34FF-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.550540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE34FF-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.550689] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.550737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.550755] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:41.628303] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14499, time_dist: generate_cluster_version=14494) [2022-08-08 11:05:41.675807] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:41.780913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:41.834428] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63D5-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:41.839273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:41.861962] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:41.861996] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667389, retry interval=1000000, ret=-4179) [2022-08-08 11:05:41.862847] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:41.862880] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667384, retry interval=1000000, ret=-4179) [2022-08-08 11:05:41.916379] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:41.931399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:41.935150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:41.941554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:41.952103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:41.967344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:42.060425] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:42.066545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:42.066738] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927942066735, cur_ts_=1659927942023557, delta=43178) [2022-08-08 11:05:42.066830] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:42.073628] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12180) [2022-08-08 11:05:42.109073] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=48342, time_dist: print_stat=6, do_cluster_heartbeat=2, generate_cluster_version=48329) [2022-08-08 11:05:42.121851] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15740, time_dist: generate_cluster_version=15733) [2022-08-08 11:05:42.188528] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:42.191711] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:42.191762] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:42.214453] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:42.235171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:42.304341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:42.360673] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:42.400881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:42.903612] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:42.903648] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667388, retry interval=1000000, ret=-4179) [2022-08-08 11:05:42.906325] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:42.906359] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667383, retry interval=1000000, ret=-4179) [2022-08-08 11:05:43.193938] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:43.198031] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:43.198082] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:43.335519] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659927943335504, cur_ts_=1659927943299588, delta=35916) [2022-08-08 11:05:43.906488] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:43.906516] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:43.908203] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63D8-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:43.928829] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:43.930233] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:43.930279] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667387, retry interval=1000000, ret=-4179) [2022-08-08 11:05:43.935306] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:43.935344] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667382, retry interval=1000000, ret=-4179) [2022-08-08 11:05:44.203366] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:44.206573] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:44.206635] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:44.546248] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927944546241, cur_ts_=1659927944519005, delta=27236) [2022-08-08 11:05:44.655956] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19821) [2022-08-08 11:05:44.656224] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21149, time_dist: do_cluster_heartbeat=11, generate_cluster_version=21135) [2022-08-08 11:05:44.934128] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D50-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:44.934174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D50-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:44.934300] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D50-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:44.934542] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D50-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:44.934687] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D50-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:44.965061] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:44.965134] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667381, retry interval=1000000, ret=-4179) [2022-08-08 11:05:44.965913] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:44.965942] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667386, retry interval=1000000, ret=-4179) [2022-08-08 11:05:45.157617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:45.209240] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:45.212475] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:45.212521] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:45.291817] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.324733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.341136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.490468] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:45.498894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=218] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.514872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.567983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.580237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.606216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:45.628798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.629378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:45.638981] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:45.681396] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.685346] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.694827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:45.695823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.696242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.707467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.707943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:45.713633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:45.718139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.750648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:45.765949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.776367] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.777040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.787708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.798036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.810935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.820435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.820498] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.820666] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.821495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.828253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.830811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.862035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.872112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.872190] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:45.873415] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.918029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.934410] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.934578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=190] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:45.993198] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:45.993245] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667380, retry interval=1000000, ret=-4179) [2022-08-08 11:05:45.996863] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:45.996947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=243] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:45.997882] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:45.997933] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667385, retry interval=1000000, ret=-4179) [2022-08-08 11:05:46.003879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:46.011557] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:46.023114] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:46.058714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:46.079249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:46.163628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:46.174080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:46.217591] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:46.220740] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:46.220794] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:46.226126] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:46.289063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:46.341229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:46.345013] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:46.588993] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE350E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:46.611924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:47.000813] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:47.023392] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:47.023429] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667384, retry interval=1000000, ret=-4179) [2022-08-08 11:05:47.028853] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:47.028918] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667379, retry interval=1000000, ret=-4179) [2022-08-08 11:05:47.222968] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:47.226356] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:47.226884] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=496] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:47.401521] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927947401515, cur_ts_=1659927947369126, delta=32389) [2022-08-08 11:05:47.988016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:48.047426] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659927948047420, cur_ts_=1659927948019317, delta=28103) [2022-08-08 11:05:48.054018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:48.054067] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:48.063333] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14172) [2022-08-08 11:05:48.063548] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16958, time_dist: do_cluster_heartbeat=8, generate_cluster_version=16949) [2022-08-08 11:05:48.077453] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14082) [2022-08-08 11:05:48.078409] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18222, time_dist: generate_cluster_version=18219) [2022-08-08 11:05:48.082039] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:48.083461] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:48.083509] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667383, retry interval=1000000, ret=-4179) [2022-08-08 11:05:48.086514] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:48.088762] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=177] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:48.088937] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667378, retry interval=1000000, ret=-4179) [2022-08-08 11:05:48.231969] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:48.235252] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:48.235289] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:48.907842] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927948907835, cur_ts_=1659927948880101, delta=27734) [2022-08-08 11:05:48.908491] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27469) [2022-08-08 11:05:48.908868] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28523, time_dist: generate_cluster_version=28520) [2022-08-08 11:05:49.113338] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:49.114518] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:49.114558] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667382, retry interval=1000000, ret=-4179) [2022-08-08 11:05:49.117314] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:49.117348] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667377, retry interval=1000000, ret=-4179) [2022-08-08 11:05:49.238002] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:49.241044] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:49.241451] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=391] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:49.485102] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15067) [2022-08-08 11:05:49.485598] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=924] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18678, time_dist: do_cluster_heartbeat=22, generate_cluster_version=18652) [2022-08-08 11:05:49.492325] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22018, time_dist: generate_cluster_version=22014) [2022-08-08 11:05:49.824987] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:05:49.825005] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:05:50.048035] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=184] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15038208}) [2022-08-08 11:05:50.164276] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:50.164326] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667381, retry interval=1000000, ret=-4179) [2022-08-08 11:05:50.173536] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:50.173971] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=314] [dc=0] task process failed, start retry(max retry time=9223372036854667376, retry interval=1000000, ret=-4179) [2022-08-08 11:05:50.245078] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:50.247943] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927948397083, end_time=1659927950247929, elapsed_time=1850846, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:05:50.250904] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:50.250943] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:50.825988] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927950825966, cur_ts_=1659927950794814, delta=31152) [2022-08-08 11:05:51.173041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:51.204096] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:51.204180] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667375, retry interval=1000000, ret=-4179) [2022-08-08 11:05:51.208480] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:51.208516] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667380, retry interval=1000000, ret=-4179) [2022-08-08 11:05:51.253643] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:51.257130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:51.257169] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:51.409104] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:51.518483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:51.529176] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10134, time_dist: generate_cluster_version=10129) [2022-08-08 11:05:51.551459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:51.630777] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE351D-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:51.630925] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE351D-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:51.631136] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:51.631319] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:51.631509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:51.692061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:51.803898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:51.862185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=250] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:51.939083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:51.953947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:51.958947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:51.964325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:51.974403] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:51.988884] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:52.082424] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:52.082483] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:52.084375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:52.206708] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C1B-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.207324] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C1B-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.207462] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C1B-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.207585] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C1B-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.208437] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.208543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.212103] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63E4-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.214643] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63E4-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:52.234423] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:52.239444] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:52.239480] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667379, retry interval=1000000, ret=-4179) [2022-08-08 11:05:52.240933] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:52.240961] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667374, retry interval=1000000, ret=-4179) [2022-08-08 11:05:52.252099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:52.259999] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:52.263635] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:52.263682] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:52.329542] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:52.379609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:52.416721] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:52.492785] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927952492781, cur_ts_=1659927952461982, delta=30799) [2022-08-08 11:05:52.555770] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10708, time_dist: do_cluster_heartbeat=9, generate_cluster_version=10695) [2022-08-08 11:05:52.559555] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11857, time_dist: do_cluster_heartbeat=8, generate_cluster_version=11845) [2022-08-08 11:05:52.693081] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:53.267410] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:53.281167] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:53.281216] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667373, retry interval=1000000, ret=-4179) [2022-08-08 11:05:53.281561] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:53.281577] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:53.287530] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:53.289194] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=85] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:53.289249] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667378, retry interval=1000000, ret=-4179) [2022-08-08 11:05:53.643535] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:53.643627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:54.284951] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:54.286980] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:54.287669] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=592] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:54.291142] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:54.291163] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:54.317728] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:54.317763] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667377, retry interval=1000000, ret=-4179) [2022-08-08 11:05:54.318930] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:54.320540] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:54.320575] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667372, retry interval=1000000, ret=-4179) [2022-08-08 11:05:55.184292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:55.296996] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:55.301245] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:55.301537] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=277] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:55.311959] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:55.334234] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10932, time_dist: generate_cluster_version=10929) [2022-08-08 11:05:55.342876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.344231] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:55.344444] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] task process failed, start retry(max retry time=9223372036854667371, retry interval=1000000, ret=-4179) [2022-08-08 11:05:55.351582] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=139] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:55.351657] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] task process failed, start retry(max retry time=9223372036854667376, retry interval=1000000, ret=-4179) [2022-08-08 11:05:55.364540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.506365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:55.520750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.531237] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.584306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.595719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=293] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:55.629856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:55.647181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.647279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:55.658365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:55.701923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:55.710706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.710804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:55.718606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:55.719352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.731665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:55.732536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:05:55.732668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=174] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:55.742941] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.773327] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:55.788060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.792700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.798033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.802927] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:55.815053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.826435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:55.836817] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.838225] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:55.838948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:55.845105] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.853161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:55.855046] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.881463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.889255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.895770] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:55.896874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:55.935338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:55.955284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:55.955338] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.017756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:05:56.020152] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:05:56.027157] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.028684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:56.047921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:56.079714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:56.121963] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.122381] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=907] [dc=0] clock out of order(cur_ts=1659927956122376, cur_ts_=1659927956097239, delta=25137) [2022-08-08 11:05:56.183651] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.193322] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.246280] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:05:56.304369] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:56.312154] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:56.312181] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:56.314894] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.348044] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.348073] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.369776] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:05:56.372907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:56.373041] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.373136] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927956373133, cur_ts_=1659927956347184, delta=25949) [2022-08-08 11:05:56.373169] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.373342] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.373364] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.373500] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.373750] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.374209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63EA-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.383011] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:56.384693] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:56.384743] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667370, retry interval=1000000, ret=-4179) [2022-08-08 11:05:56.402413] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:56.402466] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667375, retry interval=1000000, ret=-4179) [2022-08-08 11:05:56.497253] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:05:56.630552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:05:57.316431] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:57.320931] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:57.320952] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:57.404900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D64-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:57.405122] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D64-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:57.415489] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:57.417154] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:57.417204] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667369, retry interval=1000000, ret=-4179) [2022-08-08 11:05:57.424366] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:57.425922] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:57.425958] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667374, retry interval=1000000, ret=-4179) [2022-08-08 11:05:57.670467] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE352F-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.670493] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE352F-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.670845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE352F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.670865] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE352F-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.670960] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.671733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.671790] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:57.671946] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=132] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:58.012974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:05:58.324498] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659927958324488, cur_ts_=1659927958299760, delta=24728) [2022-08-08 11:05:58.325857] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:58.337470] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=1497] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:58.337497] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:58.344959] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20129) [2022-08-08 11:05:58.345659] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=48249, time_dist: generate_cluster_version=48246) [2022-08-08 11:05:58.346075] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18207, time_dist: do_cluster_heartbeat=9, generate_cluster_version=18196) [2022-08-08 11:05:58.421825] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:58.421876] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:58.464410] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:58.464445] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667368, retry interval=1000000, ret=-4179) [2022-08-08 11:05:58.469151] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:58.470628] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:58.470697] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667373, retry interval=1000000, ret=-4179) [2022-08-08 11:05:58.973572] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2787] [dc=0] clock out of order(cur_ts=1659927958973564, cur_ts_=1659927958951931, delta=21633) [2022-08-08 11:05:59.339561] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:05:59.342801] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:05:59.342829] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:05:59.500534] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:59.500572] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667372, retry interval=1000000, ret=-4179) [2022-08-08 11:05:59.501809] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18542, time_dist: do_cluster_heartbeat=3, generate_cluster_version=18537) [2022-08-08 11:05:59.509397] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:05:59.510088] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11999, time_dist: generate_cluster_version=11997) [2022-08-08 11:05:59.513175] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:05:59.513218] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667367, retry interval=1000000, ret=-4179) [2022-08-08 11:05:59.690686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3535-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:59.690703] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3535-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:59.690768] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3535-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:05:59.731373] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10248, time_dist: generate_cluster_version=10245) [2022-08-08 11:05:59.825454] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:05:59.825474] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:05:59.984675] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] clock out of order(cur_ts=1659927959984664, cur_ts_=1659927959960353, delta=24311) [2022-08-08 11:06:00.000242] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=45289) [2022-08-08 11:06:00.000895] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=46715, time_dist: generate_cluster_version=46712) [2022-08-08 11:06:00.001154] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=46029, time_dist: generate_cluster_version=46027) [2022-08-08 11:06:00.061071] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15042528}) [2022-08-08 11:06:00.111130] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659927960111122, cur_ts_=1659927960077010, delta=34112) [2022-08-08 11:06:00.278994] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927958397063, end_time=1659927960278971, elapsed_time=1881908, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:06:00.338338] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927960338327, cur_ts_=1659927960312304, delta=26023) [2022-08-08 11:06:00.344781] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:00.347750] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:00.347887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:00.471087] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659927960471078, cur_ts_=1659927960433502, delta=37576) [2022-08-08 11:06:00.507062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C27-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:00.507210] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C27-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:00.527845] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:00.529398] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:00.529432] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667371, retry interval=1000000, ret=-4179) [2022-08-08 11:06:00.538471] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:00.540013] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:00.540063] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667366, retry interval=1000000, ret=-4179) [2022-08-08 11:06:00.802627] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927960802619, cur_ts_=1659927960772287, delta=30332) [2022-08-08 11:06:01.361034] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:01.361201] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659927961361195, cur_ts_=1659927961335793, delta=25402) [2022-08-08 11:06:01.365347] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:01.365391] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:01.492232] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:01.540240] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:01.550699] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10909, time_dist: generate_cluster_version=10906) [2022-08-08 11:06:01.560896] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13768) [2022-08-08 11:06:01.565214] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23586, time_dist: generate_cluster_version=23583) [2022-08-08 11:06:01.570922] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:01.570969] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667365, retry interval=1000000, ret=-4179) [2022-08-08 11:06:01.575022] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:01.576238] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:01.576321] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667370, retry interval=1000000, ret=-4179) [2022-08-08 11:06:01.704633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:01.704820] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:01.705461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:01.711766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:01.827394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:01.881863] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:01.959857] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:01.975139] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:01.975677] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:01.984589] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:01.995349] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:02.006490] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:02.100216] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:02.103406] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:02.106609] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:02.258808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:02.267791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:02.354547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:02.367942] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:02.372200] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:02.372242] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:02.396405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:02.436448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:02.575241] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:02.575272] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:02.580582] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63F3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:02.582562] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC63F3-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:02.585812] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10772) [2022-08-08 11:06:02.601523] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:02.601570] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667364, retry interval=1000000, ret=-4179) [2022-08-08 11:06:02.606496] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:02.606578] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667369, retry interval=1000000, ret=-4179) [2022-08-08 11:06:02.717724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE353E-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.717871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE353E-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.718083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE353E-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.718210] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE353E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.718419] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.718638] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.718786] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:02.718936] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:03.374656] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:03.377586] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:03.377655] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:03.643944] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:03.644008] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667363, retry interval=1000000, ret=-4179) [2022-08-08 11:06:03.645348] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:03.645378] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667368, retry interval=1000000, ret=-4179) [2022-08-08 11:06:03.739545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3541-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:03.741649] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3541-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:03.742527] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3541-0-0] [lt=849] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:03.742543] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3541-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:03.742645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:03.742662] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:04.380200] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:04.392137] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19313) [2022-08-08 11:06:04.392410] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20738, time_dist: do_cluster_heartbeat=10, generate_cluster_version=20726) [2022-08-08 11:06:04.395694] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11579] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:04.395760] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:04.571067] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659927964571063, cur_ts_=1659927964535459, delta=35604) [2022-08-08 11:06:04.672940] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:04.672978] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667367, retry interval=1000000, ret=-4179) [2022-08-08 11:06:04.676444] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:04.676476] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667362, retry interval=1000000, ret=-4179) [2022-08-08 11:06:05.207345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:05.338872] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:05.359320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.380645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.398015] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:05.403597] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:05.403641] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:05.527187] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:05.539431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.548660] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.601667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.615756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:05.654567] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:05.666341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:05.667569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.674985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:05.676365] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.676459] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.679338] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D70-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.680629] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D70-0-0] [lt=629] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.680664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D70-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.680698] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4D70-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.698150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.701142] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:05.702295] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=1095] [dc=0] task process failed, start retry(max retry time=9223372036854667366, retry interval=1000000, ret=-4179) [2022-08-08 11:06:05.702866] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:05.704190] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:05.704241] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667361, retry interval=1000000, ret=-4179) [2022-08-08 11:06:05.721632] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:05.726900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:05.734818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.741729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:05.742301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.752935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:05.753686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:05.755644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:05.762273] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:05.762823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.795873] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:05.805613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.815650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.826556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:05.827107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.839098] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.849431] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:05.857828] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:05.859860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:05.859900] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.876205] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.878339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:05.879684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.904020] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.911882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.912394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:05.917917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:05.955405] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:05.972465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:05.974388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.004471] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927966004464, cur_ts_=1659927965975346, delta=29118) [2022-08-08 11:06:06.035475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:06.036722] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:06.047002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.047155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:06.067824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:06.096882] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:06.146597] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.202029] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.210513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.271504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:06.334184] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=1147] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.389099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=115] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:06.397478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.406386] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:06.409647] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:06.409702] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:06.626221] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659927966626213, cur_ts_=1659927966603069, delta=23144) [2022-08-08 11:06:06.649226] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12915) [2022-08-08 11:06:06.649639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:06.734865] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=576] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:06.739313] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] task process failed, start retry(max retry time=9223372036854667365, retry interval=1000000, ret=-4179) [2022-08-08 11:06:06.759917] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=3318] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15904) [2022-08-08 11:06:06.782962] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:06.782998] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667360, retry interval=1000000, ret=-4179) [2022-08-08 11:06:06.784497] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=349] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24199) [2022-08-08 11:06:06.789012] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=46152, time_dist: generate_cluster_version=46147) [2022-08-08 11:06:06.797814] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13275) [2022-08-08 11:06:07.411911] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:07.415398] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:07.415445] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:07.760283] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:07.760361] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667364, retry interval=1000000, ret=-4179) [2022-08-08 11:06:07.785005] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE354D-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.785036] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE354D-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.785297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE354D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.785462] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE354D-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.785631] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.785770] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.785966] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.786154] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:07.804735] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:07.804890] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667359, retry interval=1000000, ret=-4179) [2022-08-08 11:06:08.037868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:08.341842] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:06:08.418250] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:08.422981] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=99] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:08.423007] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:08.762822] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C33-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:08.786523] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:08.786574] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667363, retry interval=1000000, ret=-4179) [2022-08-08 11:06:08.830147] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659927968830139, cur_ts_=1659927968797221, delta=32918) [2022-08-08 11:06:08.830262] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=646] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=31225) [2022-08-08 11:06:08.830979] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=32673, time_dist: generate_cluster_version=32671) [2022-08-08 11:06:08.845650] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:08.845760] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667358, retry interval=1000000, ret=-4179) [2022-08-08 11:06:09.374226] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659927969374221, cur_ts_=1659927969351798, delta=22423) [2022-08-08 11:06:09.425912] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:09.433839] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:09.434157] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=301] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:09.811514] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:09.811635] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667362, retry interval=1000000, ret=-4179) [2022-08-08 11:06:09.828205] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:06:09.828444] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=234] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:06:09.873683] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:09.874630] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=149] [dc=0] task process failed, start retry(max retry time=9223372036854667357, retry interval=1000000, ret=-4179) [2022-08-08 11:06:10.034078] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13407) [2022-08-08 11:06:10.035534] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15650, time_dist: generate_cluster_version=15648) [2022-08-08 11:06:10.035604] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15046912}) [2022-08-08 11:06:10.219527] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927968397841, end_time=1659927970219515, elapsed_time=1821674, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:06:10.436540] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:10.442367] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:10.442394] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:10.830153] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:10.830202] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667361, retry interval=1000000, ret=-4179) [2022-08-08 11:06:10.894137] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:10.898049] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667356, retry interval=1000000, ret=-4179) [2022-08-08 11:06:11.445240] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:11.448663] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:11.448684] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:11.560841] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=561] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:11.572133] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:11.591954] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:11.735311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:11.846099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:11.859370] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26787) [2022-08-08 11:06:11.860336] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:11.860399] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667360, retry interval=1000000, ret=-4179) [2022-08-08 11:06:11.861065] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29104, time_dist: generate_cluster_version=29101) [2022-08-08 11:06:11.909389] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:11.938941] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:11.939069] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667355, retry interval=1000000, ret=-4179) [2022-08-08 11:06:11.993190] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:11.993634] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:11.993892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:11.998068] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22995, time_dist: generate_cluster_version=22990) [2022-08-08 11:06:12.003580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:12.035592] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:12.035757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:12.035837] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927972035835, cur_ts_=1659927972012392, delta=23443) [2022-08-08 11:06:12.122325] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:12.122581] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:12.124069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:12.275365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:12.288226] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:12.374684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:12.421155] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:12.454872] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:12.457912] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:12.458006] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:12.458040] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:12.693448] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:12.880497] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:12.880548] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667359, retry interval=1000000, ret=-4179) [2022-08-08 11:06:12.958687] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:12.958757] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667354, retry interval=1000000, ret=-4179) [2022-08-08 11:06:13.460762] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:13.465138] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:13.465176] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:13.912224] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:13.912260] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667358, retry interval=1000000, ret=-4179) [2022-08-08 11:06:13.991658] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:13.991720] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667353, retry interval=1000000, ret=-4179) [2022-08-08 11:06:13.992242] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=127] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21505, time_dist: do_cluster_heartbeat=9, generate_cluster_version=21495) [2022-08-08 11:06:13.992573] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11051, time_dist: generate_cluster_version=11049) [2022-08-08 11:06:14.467657] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:14.470682] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:14.470724] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:14.936457] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:14.936520] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667357, retry interval=1000000, ret=-4179) [2022-08-08 11:06:15.011752] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:15.011836] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] task process failed, start retry(max retry time=9223372036854667352, retry interval=1000000, ret=-4179) [2022-08-08 11:06:15.227007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=241] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:15.354714] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=321] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:15.375572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.405082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.473050] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:15.476321] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:15.476359] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:15.548472] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:15.557706] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.570791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.621189] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.638190] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:15.673493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:15.685185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.690526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:15.694256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:15.745072] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:15.744855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:15.755333] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.760277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.766407] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:15.774869] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:15.776136] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:15.777100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:15.783551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.811406] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:15.826782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.838061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.847626] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=159] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:15.847656] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.857976] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.874831] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:15.877321] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:15.878801] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:15.880779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.891460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:15.896137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.899837] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=121] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:15.925071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.935335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:15.936885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.936964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:15.958420] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:15.958489] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667356, retry interval=1000000, ret=-4179) [2022-08-08 11:06:15.972623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:15.988987] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:16.002824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.032780] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:16.032820] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667351, retry interval=1000000, ret=-4179) [2022-08-08 11:06:16.054545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:16.059060] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:16.065117] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.066320] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:16.089934] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:16.120953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:16.161994] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.224507] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.235913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.296156] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:16.354985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.411000] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:16.418932] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.480909] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:16.484229] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:16.484271] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:16.526342] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=174] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:06:16.666530] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=3489] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:16.981726] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:16.981781] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667355, retry interval=1000000, ret=-4179) [2022-08-08 11:06:17.051985] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:17.052020] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667350, retry interval=1000000, ret=-4179) [2022-08-08 11:06:17.154796] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] clock out of order(cur_ts=1659927977154785, cur_ts_=1659927977132266, delta=22519) [2022-08-08 11:06:17.298434] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659927977298427, cur_ts_=1659927977272546, delta=25881) [2022-08-08 11:06:17.486815] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:17.490293] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:17.490330] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:18.004879] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:18.004942] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667354, retry interval=1000000, ret=-4179) [2022-08-08 11:06:18.065692] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:18.072843] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:18.072933] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] task process failed, start retry(max retry time=9223372036854667349, retry interval=1000000, ret=-4179) [2022-08-08 11:06:18.492811] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:18.498400] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:18.498442] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:19.023456] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:19.023568] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667353, retry interval=1000000, ret=-4179) [2022-08-08 11:06:19.044096] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23340) [2022-08-08 11:06:19.044807] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26475, time_dist: generate_cluster_version=26471) [2022-08-08 11:06:19.093464] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:19.093529] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667348, retry interval=1000000, ret=-4179) [2022-08-08 11:06:19.501793] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:19.504874] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:19.504921] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:19.829344] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:06:19.829439] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:06:20.013272] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15051264}) [2022-08-08 11:06:20.051338] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:20.051404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667352, retry interval=1000000, ret=-4179) [2022-08-08 11:06:20.115139] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:20.115182] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667347, retry interval=1000000, ret=-4179) [2022-08-08 11:06:20.220309] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927978398418, end_time=1659927980220298, elapsed_time=1821880, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:06:20.507034] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:20.510481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:20.510518] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:21.073175] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:21.073295] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] task process failed, start retry(max retry time=9223372036854667351, retry interval=1000000, ret=-4179) [2022-08-08 11:06:21.137681] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:21.137889] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=110] [dc=0] task process failed, start retry(max retry time=9223372036854667346, retry interval=1000000, ret=-4179) [2022-08-08 11:06:21.514739] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:21.519335] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:21.519371] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:21.576010] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:21.609792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:21.632276] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:21.656795] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14441) [2022-08-08 11:06:21.751671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:21.864267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=65] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:21.932769] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=4403] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.012876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:22.015527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.016535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:22.023009] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:22.057330] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:22.057642] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.093611] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:22.093668] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667350, retry interval=1000000, ret=-4179) [2022-08-08 11:06:22.139527] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.141252] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:22.147028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.160522] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:22.160571] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667345, retry interval=1000000, ret=-4179) [2022-08-08 11:06:22.292952] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:22.313245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.390969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:22.437630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:22.480101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:22.527458] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:22.528271] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659927982528266, cur_ts_=1659927982508047, delta=20219) [2022-08-08 11:06:22.532698] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:22.532724] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:23.117932] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:23.117983] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667349, retry interval=1000000, ret=-4179) [2022-08-08 11:06:23.182437] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:23.182477] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667344, retry interval=1000000, ret=-4179) [2022-08-08 11:06:23.535431] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:23.539433] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:23.539459] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:24.142108] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:24.142174] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667348, retry interval=1000000, ret=-4179) [2022-08-08 11:06:24.199216] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:24.199253] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667343, retry interval=1000000, ret=-4179) [2022-08-08 11:06:24.542247] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:24.547118] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:24.547185] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:25.161709] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:25.161923] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] task process failed, start retry(max retry time=9223372036854667347, retry interval=1000000, ret=-4179) [2022-08-08 11:06:25.226033] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:25.226071] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667342, retry interval=1000000, ret=-4179) [2022-08-08 11:06:25.226600] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13770, time_dist: generate_cluster_version=13768) [2022-08-08 11:06:25.252103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:25.371387] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:25.392563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.423668] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.549680] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:25.552526] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:25.552652] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:25.568040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:25.577260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.590885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.659292] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:25.661942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.663014] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] clock out of order(cur_ts=1659927985663010, cur_ts_=1659927985635632, delta=27378) [2022-08-08 11:06:25.693439] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:25.703383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.716420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:25.726847] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:25.766648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:25.766760] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=703] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:25.773859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.777351] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.784421] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=335] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:25.794774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:25.798279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:25.799286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=324] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:25.805148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.828754] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:25.846242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3583-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.846318] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3583-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.846641] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3583-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.847057] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3583-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.847429] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.847990] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.848343] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:25.850698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.860910] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.871239] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:25.871258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.876275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.894886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:25.896943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:25.897884] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:25.903823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.913923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:25.919891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.919953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=105] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:25.940569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.950705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:25.960363] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.961867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:25.990133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.007855] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:26.017982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.078061] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:26.078357] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:26.088693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.092575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:26.114368] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:26.141697] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:26.177745] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.193650] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:26.193694] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667346, retry interval=1000000, ret=-4179) [2022-08-08 11:06:26.242038] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.249657] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=2425] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:26.250529] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=371] [dc=0] task process failed, start retry(max retry time=9223372036854667341, retry interval=1000000, ret=-4179) [2022-08-08 11:06:26.260543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.320657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:26.371178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.426719] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:26.434552] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:26.554513] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:26.559081] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:26.559137] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:26.685635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:27.221716] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:27.221753] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667345, retry interval=1000000, ret=-4179) [2022-08-08 11:06:27.272639] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:27.272710] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667340, retry interval=1000000, ret=-4179) [2022-08-08 11:06:27.569952] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:27.593067] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927987593060, cur_ts_=1659927987570898, delta=22162) [2022-08-08 11:06:27.595882] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:27.595938] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:28.082267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:28.227045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C0D-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:28.228788] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C0D-0-0] [lt=905] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:28.248136] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:28.248176] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:28.248248] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:28.250012] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:28.250074] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667344, retry interval=1000000, ret=-4179) [2022-08-08 11:06:28.297205] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:28.298078] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667339, retry interval=1000000, ret=-4179) [2022-08-08 11:06:28.600670] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:28.610299] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:28.610495] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:29.282175] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:29.282211] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667343, retry interval=1000000, ret=-4179) [2022-08-08 11:06:29.323249] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:29.323283] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667338, retry interval=1000000, ret=-4179) [2022-08-08 11:06:29.648680] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:29.648918] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=1057] [dc=0] clock out of order(cur_ts=1659927989648913, cur_ts_=1659927989619190, delta=29723) [2022-08-08 11:06:29.656883] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:29.656915] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:29.789773] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659927989789766, cur_ts_=1659927989769631, delta=20135) [2022-08-08 11:06:29.826117] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:06:29.826134] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:06:30.157479] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15055616}) [2022-08-08 11:06:30.277560] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12399) [2022-08-08 11:06:30.294110] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31183, time_dist: generate_cluster_version=31181) [2022-08-08 11:06:30.298835] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927990298827, cur_ts_=1659927990277710, delta=21117) [2022-08-08 11:06:30.331817] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:30.331873] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667342, retry interval=1000000, ret=-4179) [2022-08-08 11:06:30.347034] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:30.347168] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:30.347297] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:30.349712] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:30.349772] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667337, retry interval=1000000, ret=-4179) [2022-08-08 11:06:30.366942] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16493) [2022-08-08 11:06:30.384707] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927988399408, end_time=1659927990384694, elapsed_time=1985286, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:06:30.658963] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:30.662113] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:30.662148] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:31.341977] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.342001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.351712] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC641D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.351734] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC641D-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.351815] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC641D-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.360274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.360434] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.360470] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:31.361464] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:31.361522] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667341, retry interval=1000000, ret=-4179) [2022-08-08 11:06:31.372998] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:31.373038] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667336, retry interval=1000000, ret=-4179) [2022-08-08 11:06:31.373074] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10703) [2022-08-08 11:06:31.592755] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:31.625501] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:31.664864] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:31.670079] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:31.670107] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:31.717536] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=969] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:31.767554] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:31.885517] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3595-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.885558] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3595-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.885788] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3595-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.885844] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3595-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.885981] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.886092] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.886204] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.886309] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:31.888200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:31.956132] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=215] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.032871] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.034388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:32.039531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:32.044767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:32.073082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.077730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:32.157702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=436] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.164891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:32.169356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.312770] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=968] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:32.329092] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.366443] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659927992366439, cur_ts_=1659927992341437, delta=25002) [2022-08-08 11:06:32.375542] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C13-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.375624] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C13-0-0] [lt=47] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.375641] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C13-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.375657] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C13-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.375664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C13-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.375686] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C13-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.406219] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:32.406267] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667335, retry interval=1000000, ret=-4179) [2022-08-08 11:06:32.409006] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.409038] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.409045] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:32.410487] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:32.410521] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667340, retry interval=1000000, ret=-4179) [2022-08-08 11:06:32.415838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:32.459950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:32.502986] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:32.673492] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:32.677088] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:32.677126] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:32.693774] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:32.809588] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] clock out of order(cur_ts=1659927992809582, cur_ts_=1659927992787986, delta=21596) [2022-08-08 11:06:32.898552] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3598-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:32.898605] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3598-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:32.898851] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3598-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:32.901951] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3598-0-0] [lt=3021] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:32.902242] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:32.902283] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:32.903290] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=3729] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14934, time_dist: generate_cluster_version=14930) [2022-08-08 11:06:33.359997] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659927993359985, cur_ts_=1659927993335104, delta=24881) [2022-08-08 11:06:33.362649] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26576, time_dist: generate_cluster_version=26572) [2022-08-08 11:06:33.443020] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:33.443055] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:33.443062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:33.445099] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:33.445152] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667339, retry interval=1000000, ret=-4179) [2022-08-08 11:06:33.454245] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:33.454299] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667334, retry interval=1000000, ret=-4179) [2022-08-08 11:06:33.680520] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:33.687469] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:33.687506] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:33.790015] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659927993790009, cur_ts_=1659927993759569, delta=30440) [2022-08-08 11:06:33.909424] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:34.484328] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:34.484379] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667333, retry interval=1000000, ret=-4179) [2022-08-08 11:06:34.487730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:34.487751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:34.487757] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:34.489287] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:34.489338] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667338, retry interval=1000000, ret=-4179) [2022-08-08 11:06:34.690004] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:34.694330] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:34.694368] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:35.276803] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:35.386570] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=226] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:35.422499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.448272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.491701] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C5C-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.512256] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.512274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.512280] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.513911] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:35.513956] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667337, retry interval=1000000, ret=-4179) [2022-08-08 11:06:35.518336] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.518398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.518664] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:35.519595] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:35.519752] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667332, retry interval=1000000, ret=-4179) [2022-08-08 11:06:35.588195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:35.602233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.610888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.684301] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:35.686943] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.700540] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:35.705353] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:35.705397] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:35.718587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:35.729778] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.738790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:35.744382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:35.783080] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:35.791004] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:35.791045] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.794957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.801508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:35.811278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:35.816859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:35.823040] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:35.826208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.853671] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:35.875500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.882744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.896194] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:35.896270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.896467] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.913610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:35.916741] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:35.916795] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:35.924463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.937804] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:35.937880] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:35.944695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.959242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.966008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:35.976103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:35.986458] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:36.007283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.031551] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:36.043815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.097604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:36.100535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:36.108933] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.111007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:36.139154] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:36.158563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:36.201874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.262198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.278574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.340728] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:36.399871] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.447843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:36.457664] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.519640] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:36.519663] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:36.549916] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:36.550060] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:36.550070] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:36.551367] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:36.551416] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667331, retry interval=1000000, ret=-4179) [2022-08-08 11:06:36.554220] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=84] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:06:36.554561] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:36.554712] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667336, retry interval=1000000, ret=-4179) [2022-08-08 11:06:36.704287] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:36.708068] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:36.711269] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:36.711306] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:37.554361] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C5F-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:37.554498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C5F-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:37.554652] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C5F-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:37.557444] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6426-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:37.557967] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6426-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:37.557985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC6426-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:37.577567] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:37.577604] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667330, retry interval=1000000, ret=-4179) [2022-08-08 11:06:37.594064] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:37.595117] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=1014] [dc=0] task process failed, start retry(max retry time=9223372036854667335, retry interval=1000000, ret=-4179) [2022-08-08 11:06:37.716958] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:37.720591] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:37.720652] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:38.082196] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659927998082190, cur_ts_=1659927998058227, delta=23963) [2022-08-08 11:06:38.101479] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:38.349738] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659927998349732, cur_ts_=1659927998317674, delta=32058) [2022-08-08 11:06:38.366203] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17168) [2022-08-08 11:06:38.366455] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18860, time_dist: do_cluster_heartbeat=10, generate_cluster_version=18848) [2022-08-08 11:06:38.539440] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15288, time_dist: generate_cluster_version=15284) [2022-08-08 11:06:38.622105] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:38.622150] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667329, retry interval=1000000, ret=-4179) [2022-08-08 11:06:38.628760] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:38.631020] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:38.631055] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667334, retry interval=1000000, ret=-4179) [2022-08-08 11:06:38.634889] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10639) [2022-08-08 11:06:38.723630] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:38.728380] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:38.728418] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:38.918238] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12177, time_dist: generate_cluster_version=12173) [2022-08-08 11:06:39.660688] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:39.660722] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667328, retry interval=1000000, ret=-4179) [2022-08-08 11:06:39.661356] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:39.661567] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=102] [dc=0] task process failed, start retry(max retry time=9223372036854667333, retry interval=1000000, ret=-4179) [2022-08-08 11:06:39.731742] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:39.753224] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32563) [2022-08-08 11:06:39.753589] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=34771, time_dist: generate_cluster_version=34769) [2022-08-08 11:06:39.756030] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:39.756059] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:39.758839] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35951, time_dist: generate_cluster_version=35946) [2022-08-08 11:06:39.827989] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:06:39.828012] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:06:39.928308] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10336) [2022-08-08 11:06:39.928664] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12356, time_dist: do_cluster_heartbeat=8, generate_cluster_version=12346) [2022-08-08 11:06:39.929411] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12297, time_dist: do_cluster_heartbeat=6, generate_cluster_version=12287) [2022-08-08 11:06:40.093964] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15059936}) [2022-08-08 11:06:40.282642] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=92] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659927998400296, end_time=1659928000282630, elapsed_time=1882334, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:06:40.689227] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:40.690637] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:40.690697] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667332, retry interval=1000000, ret=-4179) [2022-08-08 11:06:40.702724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:40.702743] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:40.702821] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:40.704413] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:40.704462] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667327, retry interval=1000000, ret=-4179) [2022-08-08 11:06:40.759276] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:40.763444] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:40.763484] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:40.953945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35B0-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:40.954150] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35B0-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:40.954330] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:40.954508] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.608042] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:41.640717] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:41.706812] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC642C-0-0] [lt=352] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.706849] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC642C-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.706874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC642C-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.706885] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC642C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.707544] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC642C-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.724627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.724645] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.724705] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.726070] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:41.726116] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667331, retry interval=1000000, ret=-4179) [2022-08-08 11:06:41.727249] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.727375] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.727538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:41.729646] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:41.729749] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667326, retry interval=1000000, ret=-4179) [2022-08-08 11:06:41.772147] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:41.772712] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] clock out of order(cur_ts=1659928001772706, cur_ts_=1659928001743091, delta=29615) [2022-08-08 11:06:41.784390] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:41.784497] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:41.784515] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:41.786970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:41.906496] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:41.964696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35B3-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.964733] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35B3-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.964927] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35B3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.965122] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35B3-0-0] [lt=179] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.965296] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.965489] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=125] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:41.980946] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.053398] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659928002053385, cur_ts_=1659928002030106, delta=23279) [2022-08-08 11:06:42.053438] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:42.056433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.058191] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:42.066273] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:42.096218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:42.108465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.175907] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.190208] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:42.194696] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.330119] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:42.351102] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.434223] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:42.480328] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:42.528459] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:42.733676] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.733819] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.733922] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.734063] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.734274] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.734393] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.734541] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.734837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.735008] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.735123] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.735370] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.735460] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C22-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:42.752556] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:42.752673] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667330, retry interval=1000000, ret=-4179) [2022-08-08 11:06:42.756140] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:42.756200] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667325, retry interval=1000000, ret=-4179) [2022-08-08 11:06:42.787344] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:42.790466] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:42.790502] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:43.285718] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659928003285708, cur_ts_=1659928003258659, delta=27049) [2022-08-08 11:06:43.445966] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] clock out of order(cur_ts=1659928003445960, cur_ts_=1659928003415461, delta=30499) [2022-08-08 11:06:43.592203] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=47389) [2022-08-08 11:06:43.592427] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=56292, time_dist: generate_cluster_version=56289) [2022-08-08 11:06:43.785483] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:43.785518] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667324, retry interval=1000000, ret=-4179) [2022-08-08 11:06:43.786215] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:43.786320] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667329, retry interval=1000000, ret=-4179) [2022-08-08 11:06:43.792800] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:43.797848] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:43.797886] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:44.628503] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21039) [2022-08-08 11:06:44.634090] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27571, time_dist: generate_cluster_version=27566) [2022-08-08 11:06:44.800457] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:44.810400] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:44.810434] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:44.825213] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:44.825262] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667328, retry interval=1000000, ret=-4179) [2022-08-08 11:06:44.830943] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:44.831053] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:44.832356] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:44.832418] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667323, retry interval=1000000, ret=-4179) [2022-08-08 11:06:45.002857] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE35BC-0-0] [lt=16] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659928004999567 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659928005001001 abs_expired_time=1659928006932261 stmt_timeout=1931260 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21599716 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659928004928456 start_version=-1 trans_version=1659928004999557 commit_version=0 stmt_start_time=1659928004992601 abs_expired_time=1659928034893403 stmt_timeout=29900802 abs_lock_wait_timeout=1659928034893403 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:14521358669708481021, inc:84122998, addr:"192.168.11.128:2882", t:1659928004992601} checksum_log_ts=9223372036854775807, version=1659928004999567) [2022-08-08 11:06:45.301603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:45.407278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:45.442739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=201] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.468972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.612565] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:45.618085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.626053] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.702812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.708475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:45.739663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:45.750012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=497] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.754947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=143] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:45.760618] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:45.805766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:45.812502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.812528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:45.813620] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.814493] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:45.817661] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:45.817703] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:45.820867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:45.828735] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C6B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.828883] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C6B-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.828914] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C6B-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.828934] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C6B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.828941] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4C6B-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.830334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:45.830924] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.830940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.842580] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:45.850448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:45.852695] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.875995] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=292] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:45.893910] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.893929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.893935] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:45.895457] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:45.895491] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667327, retry interval=1000000, ret=-4179) [2022-08-08 11:06:45.898019] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:45.898049] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667322, retry interval=1000000, ret=-4179) [2022-08-08 11:06:45.900513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.902293] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.911729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.912533] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:45.914651] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.935285] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:45.935993] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:45.936232] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:45.946495] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.956962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:45.962510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:45.965068] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:45.978263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:46.004228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:46.005445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:46.005730] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:46.005807] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] clock out of order(cur_ts=1659928006005804, cur_ts_=1659928005984104, delta=21700) [2022-08-08 11:06:46.036018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.049646] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:46.059201] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.121964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:46.129529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.130649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:46.130988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:46.157259] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:46.175003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:46.175540] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=48649) [2022-08-08 11:06:46.176155] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=53761, time_dist: do_cluster_heartbeat=18, generate_cluster_version=53731) [2022-08-08 11:06:46.237212] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] clock out of order(cur_ts=1659928006237203, cur_ts_=1659928006182741, delta=54462) [2022-08-08 11:06:46.246782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.250242] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=74528) [2022-08-08 11:06:46.250407] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=11] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:16986536351103107242, inc:84123043, addr:"192.168.11.128:2882", t:1659928006129917}, log_id=8303840, used_time=110457) [2022-08-08 11:06:46.250714] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=111539, time_dist: generate_cluster_version=111537) [2022-08-08 11:06:46.280753] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.302910] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.360124] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:46.421455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.464073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:46.482133] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.725739] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:46.765356] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=130] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18389, time_dist: generate_cluster_version=18385) [2022-08-08 11:06:46.820333] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:46.823558] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:46.823615] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:46.902924] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:46.902943] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:46.905263] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:46.905299] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:46.935232] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:46.935358] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] task process failed, start retry(max retry time=9223372036854667326, retry interval=1000000, ret=-4179) [2022-08-08 11:06:46.938232] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:46.938355] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:46.939615] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:46.939666] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667321, retry interval=1000000, ret=-4179) [2022-08-08 11:06:47.675428] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659928007675421, cur_ts_=1659928007640850, delta=34571) [2022-08-08 11:06:47.828027] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=104] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:47.831704] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:47.831820] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:47.959212] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659928007959205, cur_ts_=1659928007926985, delta=32220) [2022-08-08 11:06:47.991415] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:47.991503] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:47.991510] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:47.991964] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:47.992114] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:47.992138] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:47.993429] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=287] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:47.993469] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667320, retry interval=1000000, ret=-4179) [2022-08-08 11:06:47.993615] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:47.993637] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854667325, retry interval=1000000, ret=-4179) [2022-08-08 11:06:48.111606] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C5-0-0] [lt=91] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:48.111738] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C5-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:48.112001] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C5-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:48.112034] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C5-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:48.124864] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:48.618351] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659928008618343, cur_ts_=1659928008583765, delta=34578) [2022-08-08 11:06:48.718933] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=206] [dc=0] clock out of order(cur_ts=1659928008718925, cur_ts_=1659928008674421, delta=44504) [2022-08-08 11:06:48.794816] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18163, time_dist: do_cluster_heartbeat=16, generate_cluster_version=18142) [2022-08-08 11:06:48.797464] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19550, time_dist: do_cluster_heartbeat=31, generate_cluster_version=19514) [2022-08-08 11:06:48.843629] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:48.847411] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=168] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:48.847457] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:49.043189] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:49.043257] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667324, retry interval=1000000, ret=-4179) [2022-08-08 11:06:49.054118] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:49.054209] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667319, retry interval=1000000, ret=-4179) [2022-08-08 11:06:49.117340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C8-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:49.117376] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C8-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:49.117689] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35C8-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:49.231276] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928009231270, cur_ts_=1659928009198682, delta=32588) [2022-08-08 11:06:49.411708] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=242] [dc=0] clock out of order(cur_ts=1659928009411699, cur_ts_=1659928009388383, delta=23316) [2022-08-08 11:06:49.413196] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26136, time_dist: do_cluster_heartbeat=23, generate_cluster_version=26109) [2022-08-08 11:06:49.414141] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20693, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20683) [2022-08-08 11:06:49.833103] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:06:49.833133] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:06:49.858415] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:49.861520] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:49.861558] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:50.099645] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928010099639, cur_ts_=1659928010077312, delta=22327) [2022-08-08 11:06:50.128252] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=472] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:50.128413] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667323, retry interval=1000000, ret=-4179) [2022-08-08 11:06:50.130258] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:50.130290] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667318, retry interval=1000000, ret=-4179) [2022-08-08 11:06:50.304661] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15064448}) [2022-08-08 11:06:50.490026] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928008400654, end_time=1659928010490016, elapsed_time=2089362, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:06:50.863336] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:50.866951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:50.867004] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:51.129991] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928011129986, cur_ts_=1659928011087027, delta=42959) [2022-08-08 11:06:51.203874] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15748) [2022-08-08 11:06:51.205667] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18245, time_dist: generate_cluster_version=18241) [2022-08-08 11:06:51.209185] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:51.209321] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:51.209410] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:51.210766] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:51.210844] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667322, retry interval=1000000, ret=-4179) [2022-08-08 11:06:51.215091] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:51.215127] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667317, retry interval=1000000, ret=-4179) [2022-08-08 11:06:51.629905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:51.661711] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:51.813909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:51.851274] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:51.872874] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:51.875899] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:51.875935] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:51.922572] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:52.004858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.074818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=6836] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:52.076510] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=83] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.078395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:52.088513] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:52.118036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:52.128309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.198988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.213336] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:52.213469] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC643B-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:52.213625] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC643B-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:52.219460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.260454] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:52.260490] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667316, retry interval=1000000, ret=-4179) [2022-08-08 11:06:52.262977] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:52.263022] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667321, retry interval=1000000, ret=-4179) [2022-08-08 11:06:52.355220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:52.371123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.461879] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:52.504838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:52.544578] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:52.577928] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928012577922, cur_ts_=1659928012555938, delta=21984) [2022-08-08 11:06:52.694241] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:52.878360] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:52.882768] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:52.882793] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:53.285256] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:53.285276] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:53.285281] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:53.287075] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:53.287138] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667320, retry interval=1000000, ret=-4179) [2022-08-08 11:06:53.288540] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:53.288581] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:53.290110] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:53.290155] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] task process failed, start retry(max retry time=9223372036854667315, retry interval=1000000, ret=-4179) [2022-08-08 11:06:53.885269] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:53.903867] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:53.903895] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:54.177186] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14834) [2022-08-08 11:06:54.177403] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15834, time_dist: generate_cluster_version=15832) [2022-08-08 11:06:54.291889] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC643E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:54.311074] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=114] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:54.311135] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667314, retry interval=1000000, ret=-4179) [2022-08-08 11:06:54.319769] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:54.319825] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667319, retry interval=1000000, ret=-4179) [2022-08-08 11:06:54.908597] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:54.911751] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:54.911796] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:55.176215] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:55.176400] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:55.176601] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:55.321948] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:55.344496] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:55.344639] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:55.344673] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:55.346187] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:55.346242] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667318, retry interval=1000000, ret=-4179) [2022-08-08 11:06:55.348730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:55.348863] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:55.350719] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=288] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:55.350783] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667313, retry interval=1000000, ret=-4179) [2022-08-08 11:06:55.431178] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:55.462200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.489526] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.637107] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:55.640610] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.647332] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.717838] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.731262] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:55.762502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:55.771343] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:55.772883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.777758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:55.828410] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:55.830341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.830949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:55.833859] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.840886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:55.851394] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:55.859650] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:06:55.871951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:55.872446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=192] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.899886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:55.914730] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:55.918241] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:55.918275] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:55.924505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.926181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.930465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.934540] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.936602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:55.955381] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:55.955926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:55.957906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:55.968367] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.972272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:55.978685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=589] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:55.990150] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:55.997116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=505] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:56.019902] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:56.030265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:56.030319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:56.054135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=227] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.072531] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=194] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:56.082409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.141365] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=230] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:06:56.145812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:56.153315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=3027] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:06:56.154790] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.176586] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:56.191466] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:56.270401] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=2751] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.297116] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.330648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=136] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.374172] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:56.374226] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667312, retry interval=1000000, ret=-4179) [2022-08-08 11:06:56.374862] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:56.374946] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] task process failed, start retry(max retry time=9223372036854667317, retry interval=1000000, ret=-4179) [2022-08-08 11:06:56.377298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:06:56.439598] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.489684] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:06:56.501980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.580925] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:06:56.742386] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:06:56.920551] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:56.928625] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:56.928688] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:57.410779] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13932, time_dist: generate_cluster_version=13930) [2022-08-08 11:06:57.424923] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=161] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:57.426180] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=1073] [dc=0] task process failed, start retry(max retry time=9223372036854667311, retry interval=1000000, ret=-4179) [2022-08-08 11:06:57.435091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:57.435182] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:57.435266] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:06:57.437018] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:57.437054] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667316, retry interval=1000000, ret=-4179) [2022-08-08 11:06:57.829142] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12129) [2022-08-08 11:06:57.829347] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=347] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16552, time_dist: generate_cluster_version=16551) [2022-08-08 11:06:57.934255] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:57.943707] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:57.943729] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:58.063415] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928018063398, cur_ts_=1659928018041858, delta=21540) [2022-08-08 11:06:58.149271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:06:58.206900] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35E3-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:58.206927] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35E3-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:58.207132] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35E3-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:58.207245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE35E3-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:06:58.472044] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11675, time_dist: do_cluster_heartbeat=10, generate_cluster_version=11661) [2022-08-08 11:06:58.515291] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34964) [2022-08-08 11:06:58.524705] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=5608] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=45499, time_dist: generate_cluster_version=45496) [2022-08-08 11:06:58.589286] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:58.589575] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667310, retry interval=1000000, ret=-4179) [2022-08-08 11:06:58.599280] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:58.599365] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667315, retry interval=1000000, ret=-4179) [2022-08-08 11:06:58.946289] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:58.950721] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:58.950750] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:06:59.018279] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659928019018272, cur_ts_=1659928018996245, delta=22027) [2022-08-08 11:06:59.610849] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:59.610899] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667309, retry interval=1000000, ret=-4179) [2022-08-08 11:06:59.625988] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:06:59.627148] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=703] [dc=0] task process failed, start retry(max retry time=9223372036854667314, retry interval=1000000, ret=-4179) [2022-08-08 11:06:59.826873] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:06:59.826891] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:06:59.895553] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20590) [2022-08-08 11:06:59.895959] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=22231, time_dist: generate_cluster_version=22228) [2022-08-08 11:06:59.896317] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=21899, time_dist: generate_cluster_version=21897) [2022-08-08 11:06:59.953447] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:06:59.956384] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:06:59.956421] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:00.162496] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928020162488, cur_ts_=1659928020142276, delta=20212) [2022-08-08 11:07:00.206983] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15068704}) [2022-08-08 11:07:00.395270] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928018407912, end_time=1659928020395256, elapsed_time=1987344, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:07:00.639938] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:00.639988] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667308, retry interval=1000000, ret=-4179) [2022-08-08 11:07:00.646418] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:00.646479] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667313, retry interval=1000000, ret=-4179) [2022-08-08 11:07:00.959077] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:00.962944] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:00.962973] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:01.651351] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928021651345, cur_ts_=1659928021618564, delta=32781) [2022-08-08 11:07:01.651524] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:01.699793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:01.709665] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:01.709704] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667307, retry interval=1000000, ret=-4179) [2022-08-08 11:07:01.720379] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:01.720417] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667312, retry interval=1000000, ret=-4179) [2022-08-08 11:07:01.721935] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11606, time_dist: do_cluster_heartbeat=4, generate_cluster_version=11599) [2022-08-08 11:07:01.839123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:01.911811] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=353] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:01.939256] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=233] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:01.965979] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:01.969531] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:01.969613] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:02.020810] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.091736] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.094213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:02.096261] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:02.109097] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:02.142874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:02.147298] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.220667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.231794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:02.239100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.375176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:02.394802] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.479938] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=88] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:02.528200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:02.562935] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:02.734302] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:02.734351] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667306, retry interval=1000000, ret=-4179) [2022-08-08 11:07:02.741515] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:02.741560] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667311, retry interval=1000000, ret=-4179) [2022-08-08 11:07:02.972108] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:02.976262] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:02.976311] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:03.759882] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:03.759917] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667305, retry interval=1000000, ret=-4179) [2022-08-08 11:07:03.760569] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:03.760592] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667310, retry interval=1000000, ret=-4179) [2022-08-08 11:07:03.982029] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:03.985817] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:03.985861] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:04.358793] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928024358782, cur_ts_=1659928024324813, delta=33969) [2022-08-08 11:07:04.459249] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659928024459244, cur_ts_=1659928024437203, delta=22041) [2022-08-08 11:07:04.686950] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10083, time_dist: do_cluster_heartbeat=11, generate_cluster_version=10069) [2022-08-08 11:07:04.788787] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:04.788864] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667304, retry interval=1000000, ret=-4179) [2022-08-08 11:07:04.789491] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:04.789519] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667309, retry interval=1000000, ret=-4179) [2022-08-08 11:07:04.988365] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:04.997192] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:04.997220] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:05.337947] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=154] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:05.448460] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:05.485883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.511733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.655588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:05.657071] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.662988] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.741811] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.750404] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=158] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:05.781698] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:05.792758] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:05.796123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.802950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:05.814751] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=116] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:05.814868] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667303, retry interval=1000000, ret=-4179) [2022-08-08 11:07:05.821106] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:05.821256] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] task process failed, start retry(max retry time=9223372036854667308, retry interval=1000000, ret=-4179) [2022-08-08 11:07:05.846628] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:05.846793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:05.855658] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.856926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.857121] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:05.876331] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:05.877782] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:05.888359] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.893064] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=466] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:05.923049] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:05.954220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.954729] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:05.954815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.954885] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.954975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:05.974920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:05.975465] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:05.975878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=121] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:05.985514] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.996805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:05.996878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:05.999729] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:06.002831] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:06.002868] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:06.006446] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:06.019130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:06.037563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:06.048024] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:06.050455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:06.069283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.090985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:06.103433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.163974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:06.164218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:06.172904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:06.174383] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.197261] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:06.218108] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:06.291413] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.321127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.353780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.395645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:06.460161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.512199] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:06.524033] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.765765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:06.834501] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:06.834555] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667302, retry interval=1000000, ret=-4179) [2022-08-08 11:07:06.843034] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:06.843251] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] task process failed, start retry(max retry time=9223372036854667307, retry interval=1000000, ret=-4179) [2022-08-08 11:07:07.005111] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:07.008597] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:07.008835] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=222] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:07.856907] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:07.856946] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667301, retry interval=1000000, ret=-4179) [2022-08-08 11:07:07.864068] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:07.864101] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667306, retry interval=1000000, ret=-4179) [2022-08-08 11:07:08.011537] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:08.014729] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:08.014772] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:08.173848] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:08.320948] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659928028320943, cur_ts_=1659928028285476, delta=35467) [2022-08-08 11:07:08.507107] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16816) [2022-08-08 11:07:08.507308] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17765, time_dist: do_cluster_heartbeat=10, generate_cluster_version=17753) [2022-08-08 11:07:08.549754] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17078) [2022-08-08 11:07:08.549951] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=17968, time_dist: generate_cluster_version=17966) [2022-08-08 11:07:08.883058] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=498] [dc=0] clock out of order(cur_ts=1659928028883050, cur_ts_=1659928028855807, delta=27243) [2022-08-08 11:07:08.897971] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:08.898036] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667305, retry interval=1000000, ret=-4179) [2022-08-08 11:07:08.899311] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16216, time_dist: do_cluster_heartbeat=8, generate_cluster_version=16205) [2022-08-08 11:07:08.907469] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:08.907506] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667300, retry interval=1000000, ret=-4179) [2022-08-08 11:07:08.908392] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16654) [2022-08-08 11:07:08.908604] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24759, time_dist: do_cluster_heartbeat=5, generate_cluster_version=24751) [2022-08-08 11:07:09.019026] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:09.022275] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:09.022303] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:09.510790] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659928029510782, cur_ts_=1659928029486202, delta=24580) [2022-08-08 11:07:09.538005] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27952) [2022-08-08 11:07:09.541573] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=32383, time_dist: do_cluster_heartbeat=11, generate_cluster_version=32367) [2022-08-08 11:07:09.548687] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=36248, time_dist: generate_cluster_version=36243) [2022-08-08 11:07:09.817357] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928029817348, cur_ts_=1659928029788968, delta=28380) [2022-08-08 11:07:09.828490] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:07:09.828528] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:07:09.929305] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:09.929481] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=29811) [2022-08-08 11:07:09.930860] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659928029930849, cur_ts_=1659928029910403, delta=20446) [2022-08-08 11:07:09.934086] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35562, time_dist: do_cluster_heartbeat=11, generate_cluster_version=35547) [2022-08-08 11:07:09.934303] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] task process failed, start retry(max retry time=9223372036854667304, retry interval=1000000, ret=-4179) [2022-08-08 11:07:09.955545] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:09.955706] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667299, retry interval=1000000, ret=-4179) [2022-08-08 11:07:10.024192] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:10.030934] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:10.030962] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:10.104121] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15073088}) [2022-08-08 11:07:10.290692] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928028408351, end_time=1659928030290682, elapsed_time=1882331, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:07:10.344610] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] clock out of order(cur_ts=1659928030344583, cur_ts_=1659928030312668, delta=31915) [2022-08-08 11:07:10.394096] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:07:10.954284] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16331) [2022-08-08 11:07:10.955379] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18094, time_dist: generate_cluster_version=18092) [2022-08-08 11:07:10.957229] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:10.957260] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667303, retry interval=1000000, ret=-4179) [2022-08-08 11:07:10.975147] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:10.975266] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] task process failed, start retry(max retry time=9223372036854667298, retry interval=1000000, ret=-4179) [2022-08-08 11:07:10.976674] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=29] [dc=0] [trans sampling] (TRACE=begin_ts=1659928030975678 2022-08-08 03:07:10.975678|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:6222292627174741565, inc:84123967, addr:{ip:"192.168.11.128", port:2882}, t:1659928030975000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F8131-0-0, start_time:1659928030975573, snapshot:1659928030960565|[end_task] u=331 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659928030976027|[calc_checksum_by_commit] u=19 ret:0, checksum:2862532377, checksum_log_ts:9223372036854775807|[submit_log] u=43 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=503 trans_version:1659928030975236, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:988922, logging:0, pending:0, uref:1|[end_trans_cb] u=25 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233102640|total_timeu=959) [2022-08-08 11:07:11.034543] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:11.037838] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:11.037883] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:11.260284] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE360A-0-0] [lt=6] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659928031259226 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659928031259431 abs_expired_time=1659928033189715 stmt_timeout=1930284 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21600074 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659928031168711 start_version=-1 trans_version=1659928031259216 commit_version=0 stmt_start_time=1659928031256760 abs_expired_time=1659928061157805 stmt_timeout=29901045 abs_lock_wait_timeout=1659928061157805 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:4126224143804719824, inc:84123981, addr:"192.168.11.128:2882", t:1659928031256760} checksum_log_ts=9223372036854775807, version=1659928031259226) [2022-08-08 11:07:11.668837] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=1117] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:11.726676] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:11.863604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:11.955711] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:11.983314] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:11.986466] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:11.986602] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] task process failed, start retry(max retry time=9223372036854667302, retry interval=1000000, ret=-4179) [2022-08-08 11:07:12.005229] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:12.005505] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=229] [dc=0] task process failed, start retry(max retry time=9223372036854667297, retry interval=1000000, ret=-4179) [2022-08-08 11:07:12.040633] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:12.043875] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:12.043900] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:12.044344] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.112564] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:12.116669] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.120681] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:12.127001] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:12.164630] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.168005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:12.238044] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.248962] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:12.258985] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.397843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:12.418584] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.497281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:12.544655] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:12.578905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:12.694558] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:13.007538] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:13.007572] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667301, retry interval=1000000, ret=-4179) [2022-08-08 11:07:13.027536] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11067) [2022-08-08 11:07:13.028732] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:13.028815] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667296, retry interval=1000000, ret=-4179) [2022-08-08 11:07:13.047743] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:13.051698] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:13.051733] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:13.275997] WARN [STORAGE.TRANS] wait_trans_version (ob_memtable_context.cpp:390) [10609][0][YB42C0A80B80-0005E3CFCABE3610-0-0] [lt=29] [dc=0] lock_for_read need retry(ret=-6004, *this=alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=936 is_safe_read=false read_snapshot=1659928033273820 start_version=-1 trans_version=9223372036854775807 commit_version=0 stmt_start_time=1659928033273810 abs_expired_time=1659928035204653 stmt_timeout=1930843 abs_lock_wait_timeout=0 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=true ref=0 pkey= trans_id= checksum_log_ts=0, *mem_ctx=alloc_type=0 ctx_descriptor=21600108 min_table_version=0 max_table_version=0 is_safe_read=false read_snapshot=1659928033268469 start_version=-1 trans_version=1659928033273810 commit_version=0 stmt_start_time=1659928033272395 abs_expired_time=1659928063173160 stmt_timeout=29900765 abs_lock_wait_timeout=1659928063173160 row_purge_version=0 lock_wait_start_ts=0 trx_lock_timeout=-1 end_code=0 is_readonly=false ref=0 pkey={tid:1099511627777, partition_id:0, part_cnt:1} trans_id={hash:1098257489465871292, inc:84124061, addr:"192.168.11.128:2882", t:1659928033272395} checksum_log_ts=9223372036854775807, version=1659928033273820) [2022-08-08 11:07:14.028783] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:14.028846] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667300, retry interval=1000000, ret=-4179) [2022-08-08 11:07:14.049683] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:14.049721] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667295, retry interval=1000000, ret=-4179) [2022-08-08 11:07:14.054402] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:14.059395] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:14.059440] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:15.061800] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:15.068096] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:15.068150] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667299, retry interval=1000000, ret=-4179) [2022-08-08 11:07:15.068837] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:15.068854] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:15.080566] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:15.080672] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667294, retry interval=1000000, ret=-4179) [2022-08-08 11:07:15.361433] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=200] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:15.467588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:15.502787] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:15.530536] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.676860] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:15.680607] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:15.683129] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:15.759002] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:15.770335] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:15.805556] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:15.811836] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:15.811845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.818666] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:15.864463] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:15.865478] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:15.873388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.874074] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.875027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:15.900890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:15.901445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:15.904528] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=77] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.916614] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:15.947701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:15.972982] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.978506] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:15.978794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:15.979784] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:15.980003] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:15.992284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:15.992792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:15.999892] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.010016] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:16.019164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:16.021566] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:16.029512] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:16.035913] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:16.060766] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:16.075051] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:16.075360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:16.076130] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:16.087135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.092816] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:16.092838] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:16.104805] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:16.104907] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667298, retry interval=1000000, ret=-4179) [2022-08-08 11:07:16.109175] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:16.109225] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667293, retry interval=1000000, ret=-4179) [2022-08-08 11:07:16.109397] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:16.120043] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.179377] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:16.183877] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:16.193638] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:16.198206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.219052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:16.241840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:16.314701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.341520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.377297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.413956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:16.475508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.531041] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:16.545911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:16.606464] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12733) [2022-08-08 11:07:16.606686] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13669, time_dist: generate_cluster_version=13667) [2022-08-08 11:07:16.607001] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:07:16.780973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:17.098880] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:17.102404] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:17.102428] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:17.129768] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:17.129811] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667297, retry interval=1000000, ret=-4179) [2022-08-08 11:07:17.133234] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:17.133263] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854667292, retry interval=1000000, ret=-4179) [2022-08-08 11:07:18.109379] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:18.119848] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:18.119878] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:18.161397] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:18.161510] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667296, retry interval=1000000, ret=-4179) [2022-08-08 11:07:18.169065] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:18.169111] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667291, retry interval=1000000, ret=-4179) [2022-08-08 11:07:18.191754] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:18.526954] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=93] [dc=0] clock out of order(cur_ts=1659928038526948, cur_ts_=1659928038469448, delta=57500) [2022-08-08 11:07:18.727440] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659928038727431, cur_ts_=1659928038698396, delta=29035) [2022-08-08 11:07:19.126670] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:19.133739] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:19.133765] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:19.200179] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=534] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:19.200715] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667295, retry interval=1000000, ret=-4179) [2022-08-08 11:07:19.210908] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18239, time_dist: generate_cluster_version=18234) [2022-08-08 11:07:19.219754] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20942) [2022-08-08 11:07:19.232666] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:19.232701] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667290, retry interval=1000000, ret=-4179) [2022-08-08 11:07:19.236799] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16953) [2022-08-08 11:07:19.236939] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=43012, time_dist: generate_cluster_version=43010) [2022-08-08 11:07:19.268548] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15522) [2022-08-08 11:07:19.326511] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928039326487, cur_ts_=1659928039287613, delta=38874) [2022-08-08 11:07:19.339016] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=38765) [2022-08-08 11:07:19.363667] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21503) [2022-08-08 11:07:19.468598] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=182] [dc=0] clock out of order(cur_ts=1659928039468588, cur_ts_=1659928039444972, delta=23616) [2022-08-08 11:07:19.793436] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659928039793431, cur_ts_=1659928039759960, delta=33471) [2022-08-08 11:07:19.828642] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:07:19.828748] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:07:19.911577] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=635] [dc=0] clock out of order(cur_ts=1659928039911570, cur_ts_=1659928039889139, delta=22431) [2022-08-08 11:07:20.097845] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659928040097836, cur_ts_=1659928040063461, delta=34375) [2022-08-08 11:07:20.153978] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:20.164984] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:20.165019] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:20.168076] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12921) [2022-08-08 11:07:20.168723] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14410, time_dist: print_stat=9, do_cluster_heartbeat=3, generate_cluster_version=14394) [2022-08-08 11:07:20.253448] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=546] [dc=0] clock out of order(cur_ts=1659928040253441, cur_ts_=1659928040222271, delta=31170) [2022-08-08 11:07:20.328383] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:20.328451] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] task process failed, start retry(max retry time=9223372036854667294, retry interval=1000000, ret=-4179) [2022-08-08 11:07:20.351559] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=98] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=19243) [2022-08-08 11:07:20.364092] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=3342] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:20.364292] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] task process failed, start retry(max retry time=9223372036854667289, retry interval=1000000, ret=-4179) [2022-08-08 11:07:20.377061] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11993) [2022-08-08 11:07:20.446117] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18533) [2022-08-08 11:07:20.446565] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=20366, time_dist: generate_cluster_version=20363) [2022-08-08 11:07:20.474741] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28559) [2022-08-08 11:07:20.475047] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=3289] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=47487, time_dist: generate_cluster_version=47482, cluster_service_self_check=3) [2022-08-08 11:07:20.615780] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16320) [2022-08-08 11:07:20.616234] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17530, time_dist: do_cluster_heartbeat=11, generate_cluster_version=17516) [2022-08-08 11:07:21.018623] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928041018616, cur_ts_=1659928040997806, delta=20810) [2022-08-08 11:07:21.033968] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15386) [2022-08-08 11:07:21.034976] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17608, time_dist: do_cluster_heartbeat=11, generate_cluster_version=17593) [2022-08-08 11:07:21.051052] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29655, time_dist: generate_cluster_version=29400) [2022-08-08 11:07:21.167089] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:21.172400] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:21.172443] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:21.283327] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15077472}) [2022-08-08 11:07:21.402942] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928041402937, cur_ts_=1659928041352598, delta=50339) [2022-08-08 11:07:21.408310] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=103] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32051) [2022-08-08 11:07:21.408820] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:21.410429] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] task process failed, start retry(max retry time=9223372036854667293, retry interval=1000000, ret=-4179) [2022-08-08 11:07:21.429605] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16134) [2022-08-08 11:07:21.433239] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=55519, time_dist: generate_cluster_version=55513) [2022-08-08 11:07:21.449214] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=1184] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=48917, time_dist: do_cluster_heartbeat=10, generate_cluster_version=48903) [2022-08-08 11:07:21.481337] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:21.481378] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667288, retry interval=1000000, ret=-4179) [2022-08-08 11:07:21.617428] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=59] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928038408790, end_time=1659928041617413, elapsed_time=3208623, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:07:21.617878] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11702, time_dist: generate_cluster_version=11699) [2022-08-08 11:07:21.687973] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:21.748774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:21.884437] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:21.972840] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=3496] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:22.060970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.068430] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=773] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:22.137242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:22.138590] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.138975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:22.152710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:22.178517] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:22.184993] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:22.185033] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:22.187637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:22.187701] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.261827] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.266502] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:22.281689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.422723] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:22.441444] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:22.441482] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667292, retry interval=1000000, ret=-4179) [2022-08-08 11:07:22.442868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.513156] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:22.513317] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] task process failed, start retry(max retry time=9223372036854667287, retry interval=1000000, ret=-4179) [2022-08-08 11:07:22.519878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:22.567926] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:22.596476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:22.597750] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11000) [2022-08-08 11:07:22.599078] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13321, time_dist: generate_cluster_version=13317) [2022-08-08 11:07:22.628346] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10018, time_dist: generate_cluster_version=10013) [2022-08-08 11:07:23.006615] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11501, time_dist: generate_cluster_version=11498) [2022-08-08 11:07:23.187892] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:23.191961] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:23.192001] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:23.470406] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25487) [2022-08-08 11:07:23.471888] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=476] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:23.472130] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=179] [dc=0] task process failed, start retry(max retry time=9223372036854667291, retry interval=1000000, ret=-4179) [2022-08-08 11:07:23.546054] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=61] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:23.546100] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667286, retry interval=1000000, ret=-4179) [2022-08-08 11:07:23.611790] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10215) [2022-08-08 11:07:23.612202] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11281, time_dist: generate_cluster_version=11277) [2022-08-08 11:07:23.875271] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659928043875262, cur_ts_=1659928043837494, delta=37768) [2022-08-08 11:07:24.209074] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:24.217967] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=803] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:24.218022] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:24.508543] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:24.508978] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] task process failed, start retry(max retry time=9223372036854667290, retry interval=1000000, ret=-4179) [2022-08-08 11:07:24.592393] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=682] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:24.592457] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667285, retry interval=1000000, ret=-4179) [2022-08-08 11:07:24.822640] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928044822633, cur_ts_=1659928044785747, delta=36886) [2022-08-08 11:07:24.832235] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10601, time_dist: do_cluster_heartbeat=7, generate_cluster_version=10565) [2022-08-08 11:07:25.221739] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:25.225458] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:25.225500] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:25.376621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:25.487209] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:25.519792] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:25.553980] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:25.559113] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:25.559224] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] task process failed, start retry(max retry time=9223372036854667289, retry interval=1000000, ret=-4179) [2022-08-08 11:07:25.626015] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17617) [2022-08-08 11:07:25.660253] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=934] [dc=0] clock out of order(cur_ts=1659928045660247, cur_ts_=1659928045634080, delta=26167) [2022-08-08 11:07:25.671476] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=65527, time_dist: generate_cluster_version=65523) [2022-08-08 11:07:25.690975] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=700] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:25.691046] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667284, retry interval=1000000, ret=-4179) [2022-08-08 11:07:25.694159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:25.701560] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:25.706286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:25.706885] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10617) [2022-08-08 11:07:25.780916] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:25.791603] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:25.821162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:25.831876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:25.832393] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:25.843195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:25.883052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:25.888702] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:25.897315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:25.897355] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:25.897898] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:25.918345] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:25.919700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:25.929890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:25.940255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:25.942764] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659928045942759, cur_ts_=1659928045922396, delta=20363) [2022-08-08 11:07:25.972762] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:25.994519] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=1141] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:25.996034] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:25.996282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:26.003767] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=882] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:26.004808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:26.018112] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.018173] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.022177] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:26.025643] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:26.037000] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:26.046505] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:26.046756] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:26.053665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:26.079035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:26.096374] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:26.099887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:26.110867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.130783] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:26.137547] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.214837] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:26.215069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:26.215099] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:26.215182] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.220122] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] clock out of order(cur_ts=1659928046220115, cur_ts_=1659928046181534, delta=38581) [2022-08-08 11:07:26.226458] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11343, time_dist: do_cluster_heartbeat=11, generate_cluster_version=11328) [2022-08-08 11:07:26.231386] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:26.235901] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:26.235940] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:26.241294] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:26.270177] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:26.331025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.362277] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.396746] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.437876] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:26.496931] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.560796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:26.571037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:26.571296] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928046571292, cur_ts_=1659928046544880, delta=26412) [2022-08-08 11:07:26.584260] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:26.584317] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667288, retry interval=1000000, ret=-4179) [2022-08-08 11:07:26.718836] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:26.718902] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667283, retry interval=1000000, ret=-4179) [2022-08-08 11:07:26.803956] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:27.238981] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:27.243454] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:27.243493] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:27.627872] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=131] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:27.628362] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=341] [dc=0] task process failed, start retry(max retry time=9223372036854667287, retry interval=1000000, ret=-4179) [2022-08-08 11:07:27.760114] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:27.760157] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667282, retry interval=1000000, ret=-4179) [2022-08-08 11:07:28.217206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:28.251880] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:28.255583] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:28.255634] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:28.278997] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659928048278987, cur_ts_=1659928048257599, delta=21388) [2022-08-08 11:07:28.659883] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:28.659927] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667286, retry interval=1000000, ret=-4179) [2022-08-08 11:07:28.725711] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] clock out of order(cur_ts=1659928048725700, cur_ts_=1659928048700818, delta=24882) [2022-08-08 11:07:28.810727] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:28.810809] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667281, retry interval=1000000, ret=-4179) [2022-08-08 11:07:28.897794] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12912, time_dist: generate_cluster_version=12907) [2022-08-08 11:07:29.258710] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=3242] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14114) [2022-08-08 11:07:29.259039] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15515, time_dist: generate_cluster_version=15513) [2022-08-08 11:07:29.269354] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10556) [2022-08-08 11:07:29.269629] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25335, time_dist: generate_cluster_version=25333) [2022-08-08 11:07:29.279250] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:29.282895] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:29.282923] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:29.521100] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14137) [2022-08-08 11:07:29.571713] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659928049571709, cur_ts_=1659928049547184, delta=24525) [2022-08-08 11:07:29.652355] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21525) [2022-08-08 11:07:29.653218] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23662, time_dist: generate_cluster_version=23658) [2022-08-08 11:07:29.694248] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=39370) [2022-08-08 11:07:29.698247] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:29.698318] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667285, retry interval=1000000, ret=-4179) [2022-08-08 11:07:29.704750] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=50682, time_dist: generate_cluster_version=50677) [2022-08-08 11:07:29.715435] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15537) [2022-08-08 11:07:29.834485] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21509, time_dist: do_cluster_heartbeat=15, generate_cluster_version=21491) [2022-08-08 11:07:29.838299] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:07:29.838312] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:07:29.860166] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:29.860207] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667280, retry interval=1000000, ret=-4179) [2022-08-08 11:07:30.010098] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12854, time_dist: do_cluster_heartbeat=11, generate_cluster_version=12838) [2022-08-08 11:07:30.285609] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:30.290739] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:30.290870] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:30.327026] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928050327017, cur_ts_=1659928050304764, delta=22253) [2022-08-08 11:07:30.724694] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:30.724933] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=126] [dc=0] task process failed, start retry(max retry time=9223372036854667284, retry interval=1000000, ret=-4179) [2022-08-08 11:07:30.875200] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14888) [2022-08-08 11:07:30.883495] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=24811, time_dist: generate_cluster_version=24806) [2022-08-08 11:07:30.883848] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24354, time_dist: generate_cluster_version=24351) [2022-08-08 11:07:30.890160] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:30.890202] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667279, retry interval=1000000, ret=-4179) [2022-08-08 11:07:31.073004] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21492, time_dist: do_cluster_heartbeat=10, generate_cluster_version=21477) [2022-08-08 11:07:31.086759] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14001) [2022-08-08 11:07:31.087519] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35250, time_dist: do_cluster_heartbeat=7, generate_cluster_version=35239) [2022-08-08 11:07:31.173575] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928051173566, cur_ts_=1659928051138201, delta=35365) [2022-08-08 11:07:31.293488] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:31.304666] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:31.304730] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:31.357608] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928051357598, cur_ts_=1659928051332774, delta=24824) [2022-08-08 11:07:31.483807] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11136) [2022-08-08 11:07:31.484126] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13541, time_dist: generate_cluster_version=13539) [2022-08-08 11:07:31.486607] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15081792}) [2022-08-08 11:07:31.710757] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=726] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:31.767284] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:31.797023] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659928051797013, cur_ts_=1659928051759724, delta=37289) [2022-08-08 11:07:31.859170] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:31.860030] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] task process failed, start retry(max retry time=9223372036854667283, retry interval=1000000, ret=-4179) [2022-08-08 11:07:31.860854] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=40348) [2022-08-08 11:07:31.866038] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=47365, time_dist: do_cluster_heartbeat=3, generate_cluster_version=47357) [2022-08-08 11:07:31.881471] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20549) [2022-08-08 11:07:31.935951] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:31.942607] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] clock out of order(cur_ts=1659928051942599, cur_ts_=1659928051882267, delta=60332) [2022-08-08 11:07:31.946835] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=95] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=101749, time_dist: generate_cluster_version=101745) [2022-08-08 11:07:31.954143] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=72624) [2022-08-08 11:07:31.998255] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:32.014124] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=58] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928048409602, end_time=1659928052014108, elapsed_time=3604506, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:07:32.062365] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659928052062359, cur_ts_=1659928052032053, delta=30306) [2022-08-08 11:07:32.064239] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=25760) [2022-08-08 11:07:32.081704] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:32.081739] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667278, retry interval=1000000, ret=-4179) [2022-08-08 11:07:32.083815] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.098160] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33858) [2022-08-08 11:07:32.099787] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=62282, time_dist: do_cluster_heartbeat=11, generate_cluster_version=62266) [2022-08-08 11:07:32.101014] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=39214, time_dist: do_cluster_heartbeat=8, generate_cluster_version=39202) [2022-08-08 11:07:32.102316] WARN [DETECT] ~ObDetectTimeGuard (ob_deadlock_detector_common_define.cpp:58) [10922][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] ob_lcl_batch_sender_thread.cpp:update_and_get_lcl_op_interval_() cost too much time(*this=|threshold=100us|start at 1659928052102010us|end=231us|total=231us) [2022-08-08 11:07:32.145576] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:32.147299] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=9] [dc=0] [trans sampling] (TRACE=begin_ts=1659928052146196 2022-08-08 03:07:32.146196|[init] u=0 arg1:140085232810448, ctx_type:"participant", trans_type:0, trans_id:{hash:3904136093706563530, inc:84124771, addr:{ip:"192.168.11.128", port:2882}, t:1659928052145000}, pkey:{tid:1099511628085, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC40F65DE-0-0, start_time:1659928052145988, snapshot:1659928052062599|[end_task] u=243 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659928052146457|[calc_checksum_by_commit] u=18 ret:0, checksum:568003646, checksum_log_ts:9223372036854775807|[submit_log] u=93 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=648 trans_version:1659928052145163, uref:1073741825|[exiting] u=11 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:1160985, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=3 arg1:140085232811824|total_timeu=1058) [2022-08-08 11:07:32.155078] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:32.155305] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:32.159012] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.175921] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:32.210599] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.211606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:32.285957] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=591] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:32.286158] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.303824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.325897] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:32.334330] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:32.335882] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=1512] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:32.439291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:32.464821] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.537911] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=718] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:32.585708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:32.619484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:32.695207] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:32.890205] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=361] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:32.890261] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667282, retry interval=1000000, ret=-4179) [2022-08-08 11:07:33.104104] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10542, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10502) [2022-08-08 11:07:33.126919] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:33.126959] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667277, retry interval=1000000, ret=-4179) [2022-08-08 11:07:33.342958] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:33.347379] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:33.347404] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:33.943479] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659928053943472, cur_ts_=1659928053918526, delta=24946) [2022-08-08 11:07:33.958418] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:33.958531] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667281, retry interval=1000000, ret=-4179) [2022-08-08 11:07:34.155156] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:34.155446] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] task process failed, start retry(max retry time=9223372036854667276, retry interval=1000000, ret=-4179) [2022-08-08 11:07:34.257546] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10787, time_dist: generate_cluster_version=10782) [2022-08-08 11:07:34.350670] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:34.357435] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:34.357487] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:34.580577] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659928054580569, cur_ts_=1659928054546433, delta=34136) [2022-08-08 11:07:34.985383] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:34.985445] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667280, retry interval=1000000, ret=-4179) [2022-08-08 11:07:35.191668] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:35.191754] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667275, retry interval=1000000, ret=-4179) [2022-08-08 11:07:35.365242] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:35.368883] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:35.368942] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:35.396727] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:35.517319] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:35.538260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=3227] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:35.572007] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:35.715228] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:35.721370] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=970] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:35.728598] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:35.798805] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:35.816127] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:35.842906] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:35.848856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:35.856983] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:35.866768] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:35.905896] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=1838] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:35.906414] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:35.916073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:35.921429] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:35.924992] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:35.937477] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:35.938211] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:35.947685] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:35.959164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:35.994039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=698] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:36.013161] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:36.013809] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:36.014470] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.019252] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:36.019291] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667279, retry interval=1000000, ret=-4179) [2022-08-08 11:07:36.023435] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:36.023690] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.036780] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.040917] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.042296] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:36.048574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:36.055153] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:36.062213] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.095025] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.095510] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928056095504, cur_ts_=1659928056066537, delta=28967) [2022-08-08 11:07:36.098008] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:36.099977] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.115796] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:36.118282] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.136267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.146546] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:36.157878] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.226387] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:36.226431] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667274, retry interval=1000000, ret=-4179) [2022-08-08 11:07:36.230575] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:36.230868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.230970] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:36.238817] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:36.263281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:36.288978] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:36.355858] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=489] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.374552] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:36.379085] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:36.379124] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:36.380820] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.413647] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.460909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:36.518242] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.585389] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659928056585384, cur_ts_=1659928056560240, delta=25144) [2022-08-08 11:07:36.585605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:36.596299] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.683765] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:07:36.708250] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13523, time_dist: generate_cluster_version=13518) [2022-08-08 11:07:36.718295] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18053) [2022-08-08 11:07:36.718577] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21655, time_dist: generate_cluster_version=21649, cluster_service_self_check=4) [2022-08-08 11:07:36.792367] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928056792357, cur_ts_=1659928056765668, delta=26689) [2022-08-08 11:07:36.824072] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:36.934056] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10152, time_dist: do_cluster_heartbeat=13, generate_cluster_version=10136) [2022-08-08 11:07:37.043498] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:37.043565] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667278, retry interval=1000000, ret=-4179) [2022-08-08 11:07:37.260267] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:37.260315] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667273, retry interval=1000000, ret=-4179) [2022-08-08 11:07:37.363776] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659928057363768, cur_ts_=1659928057337196, delta=26572) [2022-08-08 11:07:37.382670] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:37.384707] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11099, time_dist: do_cluster_heartbeat=6, generate_cluster_version=11087) [2022-08-08 11:07:37.388766] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15336, time_dist: print_stat=30, do_cluster_heartbeat=69, generate_cluster_version=15232) [2022-08-08 11:07:37.394452] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:37.394481] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:37.494605] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928057494595, cur_ts_=1659928057467396, delta=27199) [2022-08-08 11:07:38.081246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:38.081342] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:38.083399] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:38.083678] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=176] [dc=0] task process failed, start retry(max retry time=9223372036854667277, retry interval=1000000, ret=-4179) [2022-08-08 11:07:38.241663] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:38.264673] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:38.292367] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:38.292388] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:38.297366] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=110] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:38.297418] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667272, retry interval=1000000, ret=-4179) [2022-08-08 11:07:38.401970] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:38.408033] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:38.408062] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:38.424475] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE365B-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:38.427751] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:38.427782] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:38.607944] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] clock out of order(cur_ts=1659928058607933, cur_ts_=1659928058581757, delta=26176) [2022-08-08 11:07:39.027315] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659928059027306, cur_ts_=1659928059006315, delta=20991) [2022-08-08 11:07:39.094676] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:39.095078] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:39.164054] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=1190] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:39.166166] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=925] [dc=0] task process failed, start retry(max retry time=9223372036854667276, retry interval=1000000, ret=-4179) [2022-08-08 11:07:39.333737] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:39.335993] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:39.336080] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667271, retry interval=1000000, ret=-4179) [2022-08-08 11:07:39.414047] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:39.438548] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928059438540, cur_ts_=1659928059407235, delta=31305) [2022-08-08 11:07:39.457513] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=86] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:39.457543] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:39.468477] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE365E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:39.468518] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE365E-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:39.468534] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE365E-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:39.468623] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE365E-0-0] [lt=60] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:39.468747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:39.508160] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14408) [2022-08-08 11:07:39.508388] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=16252, time_dist: do_cluster_heartbeat=11, generate_cluster_version=16239) [2022-08-08 11:07:39.675349] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] clock out of order(cur_ts=1659928059675342, cur_ts_=1659928059653285, delta=22057) [2022-08-08 11:07:39.678821] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34871) [2022-08-08 11:07:39.851493] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:07:39.852780] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=1281] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:07:39.908344] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928059908334, cur_ts_=1659928059879014, delta=29320) [2022-08-08 11:07:39.940815] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17113) [2022-08-08 11:07:39.941067] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23235, time_dist: do_cluster_heartbeat=22, generate_cluster_version=23211) [2022-08-08 11:07:39.949612] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27442, time_dist: print_stat=9, do_cluster_heartbeat=3, generate_cluster_version=27426) [2022-08-08 11:07:40.209740] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:40.209777] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667275, retry interval=1000000, ret=-4179) [2022-08-08 11:07:40.358993] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:40.359068] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667270, retry interval=1000000, ret=-4179) [2022-08-08 11:07:40.360028] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24032) [2022-08-08 11:07:40.360804] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25621, time_dist: generate_cluster_version=25617) [2022-08-08 11:07:40.363314] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27272, time_dist: generate_cluster_version=27267) [2022-08-08 11:07:40.421297] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=7811] [dc=0] clock out of order(cur_ts=1659928060421290, cur_ts_=1659928060369713, delta=51577) [2022-08-08 11:07:40.463336] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:40.469405] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:40.469845] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=424] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:40.488631] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3661-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:40.488650] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3661-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:40.501596] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3661-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:40.501913] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3661-0-0] [lt=84] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:40.501937] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3661-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:40.501945] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3661-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:40.502085] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:41.105063] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15086176}) [2022-08-08 11:07:41.286180] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:41.286233] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667274, retry interval=1000000, ret=-4179) [2022-08-08 11:07:41.288489] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659928061284683 2022-08-08 03:07:41.284683|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:16990051374787012236, inc:84125116, addr:{ip:"192.168.11.128", port:2882}, t:1659928061283000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=8 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6C95-0-0, start_time:1659928061284481, snapshot:1659928061256991|[end_task] u=151 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659928061284855|[calc_checksum_by_commit] u=26 ret:0, checksum:4148164418, checksum_log_ts:9223372036854775807|[submit_log] u=60 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=3403 trans_version:1659928061284882, uref:1073741825|[exiting] u=15 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8304115, logging:0, pending:0, uref:1|[end_trans_cb] u=60 ret:0, param:0, param:true|[destroy] u=5 arg1:140085232375600|total_timeu=3748) [2022-08-08 11:07:41.389696] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] clock out of order(cur_ts=1659928061389690, cur_ts_=1659928061357359, delta=32331) [2022-08-08 11:07:41.393463] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4DFB-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:41.393989] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4DFB-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:41.438481] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:41.438518] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667269, retry interval=1000000, ret=-4179) [2022-08-08 11:07:41.453692] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928058410463, end_time=1659928061453678, elapsed_time=3043215, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:07:41.475049] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=74] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:41.480732] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:41.480762] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:41.628812] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] clock out of order(cur_ts=1659928061628804, cur_ts_=1659928061601660, delta=27144) [2022-08-08 11:07:41.725260] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=33513) [2022-08-08 11:07:41.726652] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=37811, time_dist: do_cluster_heartbeat=12, generate_cluster_version=37796) [2022-08-08 11:07:41.734473] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:41.743780] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=249] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11551, time_dist: generate_cluster_version=11547) [2022-08-08 11:07:41.787499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:41.848686] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=173] [dc=0] clock out of order(cur_ts=1659928061848681, cur_ts_=1659928061814163, delta=34518) [2022-08-08 11:07:41.959315] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=6606] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:42.015396] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:42.115412] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928062115401, cur_ts_=1659928062027603, delta=87798) [2022-08-08 11:07:42.124373] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.184448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:42.185448] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:42.187520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.195974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:42.225029] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27123) [2022-08-08 11:07:42.225249] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=29159, time_dist: do_cluster_heartbeat=7, generate_cluster_version=29150) [2022-08-08 11:07:42.254742] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=246] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.254904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:42.259572] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659928062259563, cur_ts_=1659928062233314, delta=26249) [2022-08-08 11:07:42.267849] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=42761) [2022-08-08 11:07:42.270915] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=73799, time_dist: do_cluster_heartbeat=8, generate_cluster_version=73787) [2022-08-08 11:07:42.291471] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:42.328606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.328712] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.329341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:42.356943] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=57973) [2022-08-08 11:07:42.391810] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=332] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:42.391889] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667273, retry interval=1000000, ret=-4179) [2022-08-08 11:07:42.407047] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21900) [2022-08-08 11:07:42.421795] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=28637, time_dist: do_cluster_heartbeat=9, generate_cluster_version=28623) [2022-08-08 11:07:42.421877] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26368, time_dist: do_cluster_heartbeat=17, generate_cluster_version=26349) [2022-08-08 11:07:42.454452] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:42.481522] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:42.481632] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:42.491059] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:42.491375] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.492289] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=130] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:42.492364] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667268, retry interval=1000000, ret=-4179) [2022-08-08 11:07:42.500505] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:42.500536] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:42.560515] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=117] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:42.608174] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:42.636765] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:43.207207] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10892, time_dist: generate_cluster_version=10889) [2022-08-08 11:07:43.376359] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10692) [2022-08-08 11:07:43.376756] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11970, time_dist: do_cluster_heartbeat=12, generate_cluster_version=11955) [2022-08-08 11:07:43.447333] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:43.447402] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:43.449818] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:43.450060] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] task process failed, start retry(max retry time=9223372036854667272, retry interval=1000000, ret=-4179) [2022-08-08 11:07:43.506082] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:43.533188] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:43.533234] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667267, retry interval=1000000, ret=-4179) [2022-08-08 11:07:43.536307] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=1350] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:43.536362] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:43.537551] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13888, time_dist: do_cluster_heartbeat=9, generate_cluster_version=13874) [2022-08-08 11:07:43.671297] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928063671287, cur_ts_=1659928063646208, delta=25079) [2022-08-08 11:07:43.785544] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928063785537, cur_ts_=1659928063760917, delta=24620) [2022-08-08 11:07:43.831039] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=9116] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=30924) [2022-08-08 11:07:43.844644] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=47789, time_dist: do_cluster_heartbeat=12, generate_cluster_version=47772) [2022-08-08 11:07:43.856578] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25450) [2022-08-08 11:07:43.856888] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=38537, time_dist: generate_cluster_version=38534) [2022-08-08 11:07:44.161095] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928064161087, cur_ts_=1659928064135457, delta=25630) [2022-08-08 11:07:44.358858] TRACE [TRACE]ob_trans_ctx.cpp:489 [10631][0][YB42C0A80B80-0005E3CF9BC835F4-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659928064350314 2022-08-08 03:07:44.350314|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:9260096169140152484, inc:84125236, addr:{ip:"192.168.11.128", port:2882}, t:1659928064349000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ACA77-0-0, start_time:1659928064350150, snapshot:1659928064306808|[end_task] u=133 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=9 stc:1659928064350465|[calc_checksum_by_commit] u=20 ret:0, checksum:2263034554, checksum_log_ts:9223372036854775807|[submit_log] u=30 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=8259 trans_version:1659928064349736, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8318728, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085233547952|total_timeu=8501) [2022-08-08 11:07:44.460974] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:44.461036] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:44.494346] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:44.494404] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667271, retry interval=1000000, ret=-4179) [2022-08-08 11:07:44.543501] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:44.547883] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:44.547926] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:44.569194] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:44.569255] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667266, retry interval=1000000, ret=-4179) [2022-08-08 11:07:44.770426] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27154) [2022-08-08 11:07:44.777916] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=36115, time_dist: generate_cluster_version=36111) [2022-08-08 11:07:44.811874] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659928064811868, cur_ts_=1659928064779498, delta=32370) [2022-08-08 11:07:44.811922] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=41442) [2022-08-08 11:07:44.822908] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=72325, time_dist: generate_cluster_version=72319) [2022-08-08 11:07:44.976699] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928064976690, cur_ts_=1659928064951175, delta=25515) [2022-08-08 11:07:44.991766] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=41622) [2022-08-08 11:07:44.992967] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=44074, time_dist: do_cluster_heartbeat=11, generate_cluster_version=44058) [2022-08-08 11:07:45.033908] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26016) [2022-08-08 11:07:45.041846] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=37022, time_dist: do_cluster_heartbeat=8, generate_cluster_version=36944) [2022-08-08 11:07:45.419185] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:45.501217] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C7E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:45.501263] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C7E-0-0] [lt=31] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:45.501282] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C7E-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:45.501292] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C7E-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:45.531774] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=1378] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:45.531876] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667270, retry interval=1000000, ret=-4179) [2022-08-08 11:07:45.533874] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:45.552209] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:45.558050] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:45.558109] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:45.563382] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:45.590415] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10600, time_dist: generate_cluster_version=10597) [2022-08-08 11:07:45.594306] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:45.603651] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:45.603693] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667265, retry interval=1000000, ret=-4179) [2022-08-08 11:07:45.738718] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=944] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:45.741748] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:45.748637] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:45.824258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:45.834621] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:45.862270] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:45.865868] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:45.874793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=687] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:45.886263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:45.925087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:45.928200] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:45.938905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:45.944083] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=3497] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:45.949207] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=1814] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:45.954072] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:45.955645] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=56] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:45.964206] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:45.976115] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:46.017794] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:46.031924] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:46.038339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:46.038395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.043248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:46.047824] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=431] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.057986] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:46.058937] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.064243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.066035] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=72] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:46.070904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:46.086867] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.114670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:46.116734] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.119021] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.136267] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:46.138975] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.160487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.168125] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:46.181964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.249891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.259309] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:46.259352] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:46.264691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:46.285360] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=129] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:46.306774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:46.378100] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.405356] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.435308] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=81] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.480037] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:46.546019] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.550557] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:46.550668] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:46.564141] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:46.580166] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:46.580215] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:46.586165] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:46.586337] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:46.588636] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:46.588954] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667269, retry interval=1000000, ret=-4179) [2022-08-08 11:07:46.601995] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=6603] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:46.614143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:46.633520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:46.633565] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:46.636662] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:46.636763] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667264, retry interval=1000000, ret=-4179) [2022-08-08 11:07:46.846085] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:47.386806] TRACE [TRACE]ob_trans_ctx.cpp:489 [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=11] [dc=0] [trans sampling] (TRACE=begin_ts=1659928067385826 2022-08-08 03:07:47.385826|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:12924628559872988260, inc:84125345, addr:{ip:"192.168.11.128", port:2882}, t:1659928067384000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=8 ret:0, left_time:10000000, uref:1073741825|[start_task] u=7 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6CB4-0-0, start_time:1659928067385661, snapshot:1659928067321820|[end_task] u=162 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=63 stc:1659928067386062|[calc_checksum_by_commit] u=26 ret:0, checksum:1589104938, checksum_log_ts:9223372036854775807|[submit_log] u=44 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=589 trans_version:1659928067386090, uref:1073741825|[exiting] u=14 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8304146, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=937) [2022-08-08 11:07:47.583712] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=654] [dc=0] clock out of order(cur_ts=1659928067583706, cur_ts_=1659928067557182, delta=26524) [2022-08-08 11:07:47.588010] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:47.598614] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3676-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:47.601887] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=323] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:47.601908] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:47.611331] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=30920, time_dist: generate_cluster_version=30916) [2022-08-08 11:07:47.683536] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:47.711699] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667268, retry interval=1000000, ret=-4179) [2022-08-08 11:07:47.711755] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:47.711773] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:47.721612] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=400] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:47.721691] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667263, retry interval=1000000, ret=-4179) [2022-08-08 11:07:47.802579] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928067802561, cur_ts_=1659928067780935, delta=21626) [2022-08-08 11:07:47.817101] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14616) [2022-08-08 11:07:47.818304] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=180] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=19352, time_dist: do_cluster_heartbeat=12, generate_cluster_version=19335) [2022-08-08 11:07:48.157074] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] clock out of order(cur_ts=1659928068157069, cur_ts_=1659928068124616, delta=32453) [2022-08-08 11:07:48.168889] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10484, time_dist: do_cluster_heartbeat=10, generate_cluster_version=10469) [2022-08-08 11:07:48.260281] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:48.582991] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12190, time_dist: generate_cluster_version=12187) [2022-08-08 11:07:48.616729] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:48.630250] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:48.630280] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:48.795339] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=4521] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=46353) [2022-08-08 11:07:48.801543] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=54873, time_dist: do_cluster_heartbeat=318, generate_cluster_version=54552) [2022-08-08 11:07:48.849365] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=53974) [2022-08-08 11:07:48.862435] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] clock out of order(cur_ts=1659928068862426, cur_ts_=1659928068817222, delta=45204) [2022-08-08 11:07:48.864516] WARN [STORAGE.TRANS] on_success (ob_trans_submit_log_cb.cpp:147) [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] transaction log sync use too much time(partition={tid:1100611139404002, partition_id:0, part_cnt:0}, trans_id={hash:5226808173314930813, inc:84125399, addr:"192.168.11.128:2882", t:1659928068759475}, log_id=8304153, used_time=103849) [2022-08-08 11:07:48.868489] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:48.879790] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] task process failed, start retry(max retry time=9223372036854667262, retry interval=1000000, ret=-4179) [2022-08-08 11:07:48.887887] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=128159, time_dist: do_cluster_heartbeat=11, generate_cluster_version=128144) [2022-08-08 11:07:48.893909] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:48.910380] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=16384] [dc=0] task process failed, start retry(max retry time=9223372036854667267, retry interval=1000000, ret=-4179) [2022-08-08 11:07:48.929953] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32357) [2022-08-08 11:07:48.963046] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21259) [2022-08-08 11:07:48.963653] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23600, time_dist: do_cluster_heartbeat=13, generate_cluster_version=23583) [2022-08-08 11:07:48.991752] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10974) [2022-08-08 11:07:48.991976] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=119] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15936, time_dist: do_cluster_heartbeat=10, generate_cluster_version=15924) [2022-08-08 11:07:49.208029] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=24386) [2022-08-08 11:07:49.211116] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=28953, time_dist: generate_cluster_version=28949) [2022-08-08 11:07:49.399060] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=13190, time_dist: generate_cluster_version=13185) [2022-08-08 11:07:49.500156] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659928069500146, cur_ts_=1659928069458862, delta=41284) [2022-08-08 11:07:49.612914] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12904) [2022-08-08 11:07:49.613186] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15083, time_dist: do_cluster_heartbeat=4, generate_cluster_version=15076) [2022-08-08 11:07:49.635369] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22376) [2022-08-08 11:07:49.635809] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27618, time_dist: generate_cluster_version=27616) [2022-08-08 11:07:49.654956] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:49.661254] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:49.661295] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:49.797179] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=75] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11795) [2022-08-08 11:07:49.800285] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=15856, time_dist: generate_cluster_version=15850) [2022-08-08 11:07:49.847026] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:07:49.847061] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:07:49.918262] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928069918245, cur_ts_=1659928069897203, delta=21042) [2022-08-08 11:07:49.952688] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:49.952725] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667261, retry interval=1000000, ret=-4179) [2022-08-08 11:07:49.961430] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:49.961559] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:49.967180] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=173] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:49.967439] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=76] [dc=0] task process failed, start retry(max retry time=9223372036854667266, retry interval=1000000, ret=-4179) [2022-08-08 11:07:49.967819] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13080) [2022-08-08 11:07:50.028312] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25147) [2022-08-08 11:07:50.038958] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=380] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12102, time_dist: generate_cluster_version=12099) [2022-08-08 11:07:50.039013] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14456, time_dist: generate_cluster_version=14456) [2022-08-08 11:07:50.224017] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=390] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=22899) [2022-08-08 11:07:50.224231] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=23877, time_dist: do_cluster_heartbeat=4, generate_cluster_version=23871) [2022-08-08 11:07:50.274060] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14023) [2022-08-08 11:07:50.277575] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18802, time_dist: generate_cluster_version=18798) [2022-08-08 11:07:50.324286] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] clock out of order(cur_ts=1659928070324281, cur_ts_=1659928070302541, delta=21740) [2022-08-08 11:07:50.474878] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] clock out of order(cur_ts=1659928070474872, cur_ts_=1659928070454825, delta=20047) [2022-08-08 11:07:50.657417] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:50.663158] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:50.667578] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:50.667611] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:50.776858] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928070776847, cur_ts_=1659928070756089, delta=20758) [2022-08-08 11:07:50.981325] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23812, time_dist: generate_cluster_version=23808) [2022-08-08 11:07:51.022477] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:51.022523] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667260, retry interval=1000000, ret=-4179) [2022-08-08 11:07:51.031997] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:51.032035] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667265, retry interval=1000000, ret=-4179) [2022-08-08 11:07:51.230495] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928071230489, cur_ts_=1659928071191894, delta=38595) [2022-08-08 11:07:51.241805] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=11420, time_dist: do_cluster_heartbeat=10, generate_cluster_version=11406) [2022-08-08 11:07:51.242574] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11132, time_dist: print_stat=5, do_cluster_heartbeat=2, generate_cluster_version=11122) [2022-08-08 11:07:51.409941] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928071409932, cur_ts_=1659928071381111, delta=28821) [2022-08-08 11:07:51.555684] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] clock out of order(cur_ts=1659928071555674, cur_ts_=1659928071528383, delta=27291) [2022-08-08 11:07:51.568977] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15090560}) [2022-08-08 11:07:51.618022] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=28771) [2022-08-08 11:07:51.618833] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=31726, time_dist: do_cluster_heartbeat=11, generate_cluster_version=31712) [2022-08-08 11:07:51.635208] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17130) [2022-08-08 11:07:51.635446] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=47032, time_dist: do_cluster_heartbeat=9, generate_cluster_version=47021) [2022-08-08 11:07:51.669870] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:51.695641] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:51.695690] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:51.756140] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:51.808856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:51.897123] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928068410956, end_time=1659928071897110, elapsed_time=3486154, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:07:51.988883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:52.029328] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E0A-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.029461] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E0A-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.029747] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E0A-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.029859] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E0A-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.035649] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=5230] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:52.050579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C87-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.051622] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C87-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.051646] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C87-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.051657] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C87-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.051668] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C87-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.051676] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C87-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.074301] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=87] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:52.074364] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667259, retry interval=1000000, ret=-4179) [2022-08-08 11:07:52.078458] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=1] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.078498] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:52.080961] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:52.080997] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667264, retry interval=1000000, ret=-4179) [2022-08-08 11:07:52.141291] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.201744] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:52.203650] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=17852) [2022-08-08 11:07:52.204053] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19396, time_dist: do_cluster_heartbeat=10, generate_cluster_version=19383) [2022-08-08 11:07:52.208286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=825] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.208605] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=472] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:52.212434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:52.276974] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:52.277082] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.345808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:52.346455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.348631] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.486069] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:52.510639] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.520822] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] clock out of order(cur_ts=1659928072520813, cur_ts_=1659928072486089, delta=34724) [2022-08-08 11:07:52.537297] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:52.585705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:52.624937] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] clock out of order(cur_ts=1659928072624929, cur_ts_=1659928072604123, delta=20806) [2022-08-08 11:07:52.629888] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:52.661563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:52.695489] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:52.698268] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:52.703708] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:52.703735] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:52.733142] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3685-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:52.733197] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3685-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:52.891099] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928072891093, cur_ts_=1659928072858062, delta=33031) [2022-08-08 11:07:53.015182] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=442] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14516, time_dist: generate_cluster_version=14511) [2022-08-08 11:07:53.154486] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=2673] [dc=0] clock out of order(cur_ts=1659928073154478, cur_ts_=1659928073122568, delta=31910) [2022-08-08 11:07:53.199484] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10465) [2022-08-08 11:07:53.201109] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=1624] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:53.204815] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=138] [dc=0] task process failed, start retry(max retry time=9223372036854667263, retry interval=1000000, ret=-4179) [2022-08-08 11:07:53.204863] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18878, time_dist: generate_cluster_version=18876) [2022-08-08 11:07:53.206470] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:53.206520] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:53.209896] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:53.209932] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667258, retry interval=1000000, ret=-4179) [2022-08-08 11:07:53.213816] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=26587, time_dist: do_cluster_heartbeat=11, generate_cluster_version=26571) [2022-08-08 11:07:53.600203] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=21] [dc=0] [trans sampling] (TRACE=begin_ts=1659928073599127 2022-08-08 03:07:53.599127|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:18036613241531312353, inc:84125579, addr:{ip:"192.168.11.128", port:2882}, t:1659928073598000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=5 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6CD3-0-0, start_time:1659928073598997, snapshot:1659928073550033|[end_task] u=127 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659928073599270|[calc_checksum_by_commit] u=21 ret:0, checksum:1020229109, checksum_log_ts:9223372036854775807|[submit_log] u=54 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=764 trans_version:1659928073599292, uref:1073741825|[exiting] u=25 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=2 ret:0, log_type:64, log_id:8304177, logging:0, pending:0, uref:1|[end_trans_cb] u=16 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1033) [2022-08-08 11:07:53.706681] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:53.717679] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:53.718414] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=715] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:53.742206] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=190] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.742240] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.742246] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.742251] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.742270] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.742968] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.743017] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=35] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.744410] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.744694] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.745018] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.745052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3688-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.745177] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.745195] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.745208] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:53.745340] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:54.209720] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:54.209844] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:54.250485] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:54.250526] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667262, retry interval=1000000, ret=-4179) [2022-08-08 11:07:54.263933] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:54.263971] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667257, retry interval=1000000, ret=-4179) [2022-08-08 11:07:54.738014] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:54.739781] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928074739727, cur_ts_=1659928074717675, delta=22052) [2022-08-08 11:07:54.752250] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:54.752311] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:54.760874] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE368B-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:55.098255] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] clock out of order(cur_ts=1659928075098246, cur_ts_=1659928075069665, delta=28581) [2022-08-08 11:07:55.285578] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=123] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:55.285724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:55.287591] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:55.287672] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] task process failed, start retry(max retry time=9223372036854667261, retry interval=1000000, ret=-4179) [2022-08-08 11:07:55.303233] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:55.303276] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667256, retry interval=1000000, ret=-4179) [2022-08-08 11:07:55.434800] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=121] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:55.552484] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:55.586055] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:55.612388] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:55.757161] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:55.760568] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:55.763857] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:55.764007] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=124] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:55.764220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:55.764236] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:55.775538] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:55.842679] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:55.854919] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:55.880219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:55.891353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:55.894667] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:55.903047] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:55.944436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=79] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:55.947258] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:55.955602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=354] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:55.964887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:55.968052] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:55.976036] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:07:55.976705] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:55.987279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:55.994762] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:56.041482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:56.056591] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.057779] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:56.063006] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:56.063176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.064260] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:56.074708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.078942] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:56.083476] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.085066] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:56.090543] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=97] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:56.106017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.132137] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:56.137101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.145574] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.152998] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:56.161623] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=152] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.176961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.187491] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=314] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:56.197600] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=128] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.273101] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.280969] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:56.281413] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:07:56.282475] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=187] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:07:56.291359] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E10-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:56.293212] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:56.302276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:56.331856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:56.347633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:56.347679] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:56.352647] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:56.352687] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667255, retry interval=1000000, ret=-4179) [2022-08-08 11:07:56.360409] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=1243] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:56.362495] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:56.365044] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:56.365084] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667260, retry interval=1000000, ret=-4179) [2022-08-08 11:07:56.377360] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10507) [2022-08-08 11:07:56.399149] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.422978] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.473541] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] clock out of order(cur_ts=1659928076473534, cur_ts_=1659928076442780, delta=30754) [2022-08-08 11:07:56.475114] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.504276] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:07:56.563629] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.622672] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:07:56.640279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:56.653975] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] [trans sampling] (TRACE=begin_ts=1659928076652310 2022-08-08 03:07:56.652310|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:3661059220690956698, inc:84125693, addr:{ip:"192.168.11.128", port:2882}, t:1659928076651000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=42 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ACAB5-0-0, start_time:1659928076651938, snapshot:1659928076624202|[end_task] u=606 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=12 stc:1659928076652972|[calc_checksum_by_commit] u=23 ret:0, checksum:2022331808, checksum_log_ts:9223372036854775807|[submit_log] u=32 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=0 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=850 trans_version:1659928076652932, uref:1073741825|[exiting] u=25 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8318790, logging:0, pending:0, uref:1|[end_trans_cb] u=19 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=1621) [2022-08-08 11:07:56.716782] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:07:56.767302] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:56.773563] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:56.773590] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:56.801496] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3691-0-0] [lt=81] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:56.802626] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3691-0-0] [lt=29] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:56.803995] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3691-0-0] [lt=1335] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:56.804258] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:56.804281] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:56.804294] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:56.862253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:07:57.048359] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=588] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10009) [2022-08-08 11:07:57.049037] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11511, time_dist: do_cluster_heartbeat=8, generate_cluster_version=11493) [2022-08-08 11:07:57.358871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:57.358897] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:57.403567] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:57.403610] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667259, retry interval=1000000, ret=-4179) [2022-08-08 11:07:57.411742] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:57.415499] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:57.415743] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] task process failed, start retry(max retry time=9223372036854667254, retry interval=1000000, ret=-4179) [2022-08-08 11:07:57.664259] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=294] [dc=0] clock out of order(cur_ts=1659928077664237, cur_ts_=1659928077635692, delta=28545) [2022-08-08 11:07:57.781324] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:57.785510] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:57.785562] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:57.817216] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE3694-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:57.819100] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:07:57.881617] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21528) [2022-08-08 11:07:57.882226] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23049, time_dist: generate_cluster_version=23045) [2022-08-08 11:07:58.252307] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] clock out of order(cur_ts=1659928078252298, cur_ts_=1659928078214057, delta=38241) [2022-08-08 11:07:58.265420] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11523, time_dist: do_cluster_heartbeat=7, generate_cluster_version=11511) [2022-08-08 11:07:58.323312] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:07:58.434326] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=225] [dc=0] clock out of order(cur_ts=1659928078434316, cur_ts_=1659928078388352, delta=45964) [2022-08-08 11:07:58.487934] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=43099) [2022-08-08 11:07:58.511930] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=70951, time_dist: do_cluster_heartbeat=17, generate_cluster_version=70929) [2022-08-08 11:07:58.512824] WARN [CLOG] submit_log_succ_cb (ob_log_task.cpp:323) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=24] [dc=0] transaction on success costs(on_success_duration=24814, pkey={tid:1099511628002, partition_id:0, part_cnt:0}, log_id=8318799) [2022-08-08 11:07:58.512938] WARN [CLOG] flush_cb (ob_partition_log_service.cpp:3442) [10632][0][YB42C0A80B80-0005E3CF9BC835FC-0-0] [lt=71] [dc=0] flush_cb cost too much time(partition_key_={tid:1099511628002, partition_id:0, part_cnt:0}, arg={log_type:201, log_id:8318799, proposal_id:{time_to_usec:1659815620290075, server:"192.168.11.128:2882"}, leader:"0.0.0.0", cluster_id:1, log_cursor:{file_id:237, offset:56586208, size:569}, after_consume_timestamp:1659928078483717}, end_time=1659928078510907, begin_time=1659928078483717, cost_time=27190) [2022-08-08 11:07:58.513261] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25281) [2022-08-08 11:07:58.550611] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=104604, time_dist: do_cluster_heartbeat=13, generate_cluster_version=104587) [2022-08-08 11:07:58.550739] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] clock out of order(cur_ts=1659928078550733, cur_ts_=1659928078514403, delta=36330) [2022-08-08 11:07:58.641364] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:58.641403] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667253, retry interval=1000000, ret=-4179) [2022-08-08 11:07:58.661431] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:58.661478] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] task process failed, start retry(max retry time=9223372036854667258, retry interval=1000000, ret=-4179) [2022-08-08 11:07:58.665528] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20304) [2022-08-08 11:07:58.683684] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10012, time_dist: do_cluster_heartbeat=11, generate_cluster_version=9994) [2022-08-08 11:07:58.792222] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:58.819951] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:58.820076] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:59.452746] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13548) [2022-08-08 11:07:59.453405] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=19064, time_dist: generate_cluster_version=19062) [2022-08-08 11:07:59.678526] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:59.678566] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667252, retry interval=1000000, ret=-4179) [2022-08-08 11:07:59.699368] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=409] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11365) [2022-08-08 11:07:59.708174] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:07:59.710414] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=231] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:07:59.710793] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=290] [dc=0] task process failed, start retry(max retry time=9223372036854667257, retry interval=1000000, ret=-4179) [2022-08-08 11:07:59.823651] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:07:59.827583] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:07:59.827612] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:07:59.867578] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:07:59.867599] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:08:00.057448] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12006, time_dist: do_cluster_heartbeat=10, generate_cluster_version=11990) [2022-08-08 11:08:00.264905] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13636) [2022-08-08 11:08:00.265911] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20672, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20661) [2022-08-08 11:08:00.414213] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659928080414208, cur_ts_=1659928080389882, delta=24326) [2022-08-08 11:08:00.645525] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] clock out of order(cur_ts=1659928080645516, cur_ts_=1659928080625142, delta=20374) [2022-08-08 11:08:00.795521] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:00.798943] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=511] [dc=0] task process failed, start retry(max retry time=9223372036854667251, retry interval=1000000, ret=-4179) [2022-08-08 11:08:00.799837] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:00.799980] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=125] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:00.802863] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:00.802903] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667256, retry interval=1000000, ret=-4179) [2022-08-08 11:08:00.856588] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928080856574, cur_ts_=1659928080816492, delta=40082) [2022-08-08 11:08:00.858183] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:00.896720] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=486] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=37339) [2022-08-08 11:08:00.897309] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=47653, time_dist: print_stat=9, do_cluster_heartbeat=3, generate_cluster_version=47637) [2022-08-08 11:08:00.897668] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:00.897710] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:00.907092] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=55556, time_dist: do_cluster_heartbeat=12, generate_cluster_version=55539) [2022-08-08 11:08:01.116261] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10579, time_dist: generate_cluster_version=10574) [2022-08-08 11:08:01.238235] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=352] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15094816}) [2022-08-08 11:08:01.319668] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=109] [dc=0] clock out of order(cur_ts=1659928081319659, cur_ts_=1659928081297285, delta=22374) [2022-08-08 11:08:01.320313] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27771) [2022-08-08 11:08:01.323428] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=31659, time_dist: do_cluster_heartbeat=9, generate_cluster_version=31645) [2022-08-08 11:08:01.336861] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16320) [2022-08-08 11:08:01.337197] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=46451, time_dist: do_cluster_heartbeat=72, generate_cluster_version=46375) [2022-08-08 11:08:01.453686] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16943) [2022-08-08 11:08:01.454256] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=166] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=18427, time_dist: generate_cluster_version=18423) [2022-08-08 11:08:01.458738] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10190, time_dist: generate_cluster_version=10183) [2022-08-08 11:08:01.677307] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=111] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928078413228, end_time=1659928081677293, elapsed_time=3264065, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:08:01.779891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:01.824051] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=460] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:01.865849] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:01.866411] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=122] [dc=0] task process failed, start retry(max retry time=9223372036854667250, retry interval=1000000, ret=-4179) [2022-08-08 11:08:01.868835] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=757] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:01.868889] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667255, retry interval=1000000, ret=-4179) [2022-08-08 11:08:01.908877] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:01.912345] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:01.912384] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:02.007254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:02.060613] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:02.161928] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.226612] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=96] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:02.227665] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.230817] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=67] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:02.237563] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:02.299576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:02.304818] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=128] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.305210] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=170] [dc=0] clock out of order(cur_ts=1659928082305206, cur_ts_=1659928082282151, delta=23055) [2022-08-08 11:08:02.309864] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=27303) [2022-08-08 11:08:02.316152] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=36094, time_dist: generate_cluster_version=36090) [2022-08-08 11:08:02.363488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=158] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:02.369950] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.371774] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.508561] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=800] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:02.527181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.608311] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:02.613880] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:02.647167] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:02.681238] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=43] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:02.894272] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=17915, time_dist: generate_cluster_version=17913) [2022-08-08 11:08:02.925693] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:02.926740] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=32279) [2022-08-08 11:08:02.938356] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=55474, time_dist: print_stat=7, generate_cluster_version=55464) [2022-08-08 11:08:02.944410] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=101] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:02.945986] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=1484] [dc=0] task process failed, start retry(max retry time=9223372036854667249, retry interval=1000000, ret=-4179) [2022-08-08 11:08:02.949539] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:02.949578] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:02.965633] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:02.965681] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:02.969094] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:02.969135] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667254, retry interval=1000000, ret=-4179) [2022-08-08 11:08:03.619321] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=11220) [2022-08-08 11:08:03.619728] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12514, time_dist: generate_cluster_version=12505, cluster_service_self_check=6) [2022-08-08 11:08:03.692583] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=278] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14082, time_dist: generate_cluster_version=14077) [2022-08-08 11:08:03.760247] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] clock out of order(cur_ts=1659928083760241, cur_ts_=1659928083734170, delta=26071) [2022-08-08 11:08:03.893088] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=50] [dc=0] clock out of order(cur_ts=1659928083893083, cur_ts_=1659928083861193, delta=31890) [2022-08-08 11:08:03.955382] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:03.962958] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:03.963003] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:04.001116] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928084001109, cur_ts_=1659928083971995, delta=29114) [2022-08-08 11:08:04.067547] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:04.067612] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667253, retry interval=1000000, ret=-4179) [2022-08-08 11:08:04.071759] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:04.071790] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667248, retry interval=1000000, ret=-4179) [2022-08-08 11:08:04.679612] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12676, time_dist: generate_cluster_version=12671) [2022-08-08 11:08:05.008552] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:05.013569] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=181] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:05.013662] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=78] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:05.015895] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36A9-0-0] [lt=83] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:05.016209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36A9-0-0] [lt=267] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:05.083091] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E1C-0-0] [lt=112] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.083931] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E1C-0-0] [lt=781] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.083952] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E1C-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.083963] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E1C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.084062] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E1C-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.141331] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=60531) [2022-08-08 11:08:05.141662] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=37] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=65458, time_dist: generate_cluster_version=65453, cluster_service_self_check=3) [2022-08-08 11:08:05.175732] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=34276) [2022-08-08 11:08:05.176594] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928085176588, cur_ts_=1659928085153916, delta=22672) [2022-08-08 11:08:05.198197] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=90433, time_dist: generate_cluster_version=90428) [2022-08-08 11:08:05.220245] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.220323] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:05.224473] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:05.224536] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667252, retry interval=1000000, ret=-4179) [2022-08-08 11:08:05.235134] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:05.235192] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667247, retry interval=1000000, ret=-4179) [2022-08-08 11:08:05.271836] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15084, time_dist: do_cluster_heartbeat=10, generate_cluster_version=15067) [2022-08-08 11:08:05.279181] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=8447] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=20091, time_dist: do_cluster_heartbeat=8, generate_cluster_version=20078) [2022-08-08 11:08:05.366823] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] clock out of order(cur_ts=1659928085366810, cur_ts_=1659928085334718, delta=32092) [2022-08-08 11:08:05.457469] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:05.574032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:05.604700] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:05.629390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:05.781793] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:05.783313] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:05.785263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:05.865234] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:05.876604] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:05.900775] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:05.907724] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:05.919162] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:05.929103] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:05.963798] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:05.966341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:05.977789] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:05.988229] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:05.988494] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=80] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:05.994409] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:05.999123] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:06.009689] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.019221] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:06.020949] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=211] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:06.024314] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:06.024361] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:06.032021] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36AC-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:06.032074] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36AC-0-0] [lt=45] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:06.062253] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:06.072181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.079248] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:06.081244] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:06.081733] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:06.082853] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.091923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.102039] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=253] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:06.104272] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:06.104339] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.112135] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:06.124545] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=89] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.149791] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=113] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:06.155899] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.164048] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=167] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.174617] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:06.184893] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.192835] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=69] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.207219] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:06.217799] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.227561] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.227616] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.230550] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.230579] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.230593] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.231173] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.231198] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.231209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.231230] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.231424] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CDD-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.232558] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.232577] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:06.257030] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:06.257080] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667246, retry interval=1000000, ret=-4179) [2022-08-08 11:08:06.257607] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:06.257632] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667251, retry interval=1000000, ret=-4179) [2022-08-08 11:08:06.294000] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.302148] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:06.303445] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:06.304434] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:06.322967] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:06.354891] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:06.417028] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.439856] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=250] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.492923] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.527181] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:06.589399] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=54] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.647159] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:06.661138] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:06.879283] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:07.030147] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:07.034278] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:07.034304] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:07.040739] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:07.261292] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:07.287013] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=1681] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:07.287033] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:07.297803] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:07.297864] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667245, retry interval=1000000, ret=-4179) [2022-08-08 11:08:07.298684] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:07.298725] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667250, retry interval=1000000, ret=-4179) [2022-08-08 11:08:08.037165] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:08.040635] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:08.040661] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:08.047696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36B2-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.047724] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36B2-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.047965] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36B2-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.048189] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36B2-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.048531] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.048852] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.049046] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.049696] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:08.302068] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE0-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:08.302106] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE0-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:08.302209] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE0-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:08.324028] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] clock out of order(cur_ts=1659928088324024, cur_ts_=1659928088301717, delta=22307) [2022-08-08 11:08:08.330561] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:08.330617] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667244, retry interval=1000000, ret=-4179) [2022-08-08 11:08:08.336151] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:08.336185] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:08.337517] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:08.337570] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667249, retry interval=1000000, ret=-4179) [2022-08-08 11:08:08.344249] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:08.509897] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12319) [2022-08-08 11:08:08.510101] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13918, time_dist: generate_cluster_version=13915) [2022-08-08 11:08:08.850950] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659928088850945, cur_ts_=1659928088820800, delta=30145) [2022-08-08 11:08:08.928000] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25714) [2022-08-08 11:08:08.928188] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=27306, time_dist: generate_cluster_version=27305) [2022-08-08 11:08:09.053250] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:09.060676] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] clock out of order(cur_ts=1659928089060668, cur_ts_=1659928089036775, delta=23893) [2022-08-08 11:08:09.069032] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12742, time_dist: generate_cluster_version=12737) [2022-08-08 11:08:09.073104] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:09.073132] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:09.178582] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928089178573, cur_ts_=1659928089155660, delta=22913) [2022-08-08 11:08:09.302434] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12483) [2022-08-08 11:08:09.302643] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13410, time_dist: print_stat=9, generate_cluster_version=13397) [2022-08-08 11:08:09.333398] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][YB42C0A80B80-0005E3CFC60C4E22-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:09.334509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:09.334656] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:09.342328] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C9F-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:09.342352] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4C9F-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:09.372940] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=20658] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:09.373002] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667243, retry interval=1000000, ret=-4179) [2022-08-08 11:08:09.375379] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928089375375, cur_ts_=1659928089351103, delta=24272) [2022-08-08 11:08:09.388407] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:09.388471] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667248, retry interval=1000000, ret=-4179) [2022-08-08 11:08:09.850646] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:08:09.850689] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:08:09.893932] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10335) [2022-08-08 11:08:09.894236] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11335, time_dist: generate_cluster_version=11332) [2022-08-08 11:08:10.074604] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:10.075359] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10622][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:10.075396] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10622][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x6d3369c 0x3cd322f 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:10.075406] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15099200}) [2022-08-08 11:08:10.261550] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928088414597, end_time=1659928090261538, elapsed_time=1846941, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:08:10.390562] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE3-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:10.396316] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:10.396353] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667242, retry interval=1000000, ret=-4179) [2022-08-08 11:08:10.410271] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:10.410368] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:10.411721] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:10.411782] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] task process failed, start retry(max retry time=9223372036854667247, retry interval=1000000, ret=-4179) [2022-08-08 11:08:11.082621] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:11.085689] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:11.085731] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:11.419408] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CA2-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:11.435107] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:11.435158] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667241, retry interval=1000000, ret=-4179) [2022-08-08 11:08:11.440095] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:11.441667] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=49] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:11.441721] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667246, retry interval=1000000, ret=-4179) [2022-08-08 11:08:11.796862] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:11.848953] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.034555] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] clock out of order(cur_ts=1659928092034548, cur_ts_=1659928092005128, delta=29420) [2022-08-08 11:08:12.035005] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.085708] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:12.090550] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:12.093750] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:12.093795] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:12.166342] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=12954, time_dist: generate_cluster_version=12950) [2022-08-08 11:08:12.177271] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=202] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.247417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.248198] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=68] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.248743] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.257245] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.321587] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:12.330250] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.383754] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:12.392674] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.393870] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=57] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.438182] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][YB42C0A80B80-0005E3CFC5EC64B1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:12.470028] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:12.470086] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667240, retry interval=1000000, ret=-4179) [2022-08-08 11:08:12.474749] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:12.474911] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] task process failed, start retry(max retry time=9223372036854667245, retry interval=1000000, ret=-4179) [2022-08-08 11:08:12.480878] WARN [SERVER] init_lease_request (ob_heartbeat.cpp:154) [10926][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] error occurs on data disk or slog disk(is_data_disk_error=true, is_slog_disk_warning=false) [2022-08-08 11:08:12.532464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.544032] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=274] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.631633] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:12.672648] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=251] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:12.692409] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=70] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:12.695808] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10623][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:12.703750] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:13.096377] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:13.099417] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:13.099455] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:13.473102] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:13.494651] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:13.494718] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667239, retry interval=1000000, ret=-4179) [2022-08-08 11:08:13.504591] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:13.504614] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:13.506031] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:13.506069] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667244, retry interval=1000000, ret=-4179) [2022-08-08 11:08:13.545768] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] clock out of order(cur_ts=1659928093545757, cur_ts_=1659928093515005, delta=30752) [2022-08-08 11:08:14.101771] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:14.107596] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:14.107682] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:14.139469] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36C4-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:14.139537] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36C4-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:14.139770] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36C4-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:14.139950] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36C4-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:14.140131] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:14.140410] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:14.508608] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=21] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.508997] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=90] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.509260] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=159] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.509516] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=227] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.509678] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=142] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.509719] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.509871] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CE9-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.525119] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:14.525165] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667238, retry interval=1000000, ret=-4179) [2022-08-08 11:08:14.530612] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.530661] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:14.532183] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=82] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:14.532326] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] task process failed, start retry(max retry time=9223372036854667243, retry interval=1000000, ret=-4179) [2022-08-08 11:08:14.706130] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=10457, time_dist: generate_cluster_version=10452) [2022-08-08 11:08:14.706627] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=10142, time_dist: generate_cluster_version=10139) [2022-08-08 11:08:15.110171] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:15.113876] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:15.114455] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=561] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:15.475195] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:15.530283] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.530470] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=165] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.534924] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CA8-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.534953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CA8-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.534975] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CA8-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.534985] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CA8-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.554143] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:15.554191] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667237, retry interval=1000000, ret=-4179) [2022-08-08 11:08:15.555953] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.556083] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:15.558930] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:15.559079] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667242, retry interval=1000000, ret=-4179) [2022-08-08 11:08:15.590464] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:15.625961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:15.652747] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:15.804113] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=671] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:15.805670] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:15.808378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:15.906624] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:15.906672] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659928095906667, cur_ts_=1659928095881011, delta=25656) [2022-08-08 11:08:15.906693] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:15.918504] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:15.929243] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:15.934842] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:15.949499] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:15.981059] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:15.991334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:16.001535] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.005395] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.012147] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.012275] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.022378] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=742] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:16.032904] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=90] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.044176] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:16.084997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:16.095725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:16.097785] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.097966] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.105788] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:16.105961] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:16.108172] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.118680] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:16.121244] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:16.123080] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=214] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:16.123207] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=112] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:16.127482] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:16.127576] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:16.130297] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.149286] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.170114] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:16.178972] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.187500] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.200057] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.200164] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:16.211809] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.231278] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:16.236569] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.314027] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=1034] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.322218] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:16.323265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:16.325455] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:16.343710] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:16.373480] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:16.437909] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.457324] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.509390] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.550548] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=279] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:16.560852] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CEC-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:16.561395] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CEC-0-0] [lt=136] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:16.561706] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CEC-0-0] [lt=9] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:16.562050] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][YB42C0A80B80-0005E3CFC61C4CEC-0-0] [lt=8] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:16.591204] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:16.591244] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667236, retry interval=1000000, ret=-4179) [2022-08-08 11:08:16.593891] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=21] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:16.593923] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667241, retry interval=1000000, ret=-4179) [2022-08-08 11:08:16.609018] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.665691] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:16.681764] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:16.739857] WARN [STORAGE.TRANS] can_be_minor_merged (ob_memtable.cpp:2159) [10581][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] memtable cannot be minor merged(*this={ObITable:{this:0x7f683a07f340, key:{table_type:0, pkey:{tid:1099511627971, partition_id:14, part_cnt:0}, table_id:1099511627971, trans_version_range:{multi_version_start:1659874663407947, base_version:1659874663407947, snapshot_version:9223372036854775807}, log_ts_range:{start_log_ts:1659874663407947, end_log_ts:9223372036854775807, max_log_ts:0}, version:"0-0-0"}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1659874663498773}, this:0x7f683a07f340, timestamp:1659874663498773, active_trx_count:0, state:0, max_schema_version:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:0, id:-1, clock:9223372036854775807}, host:0x7f69a8d51730, arena_handle:{allocated:0}, last_freeze_timestamp:1657855630002491}, with_accurate_log_ts_range:true, frozen:false, frozen_log_applied:false, mark_finish:false, pending_cb_cnt:0, pending_lob_cnt:0, pending_batch_commit_cnt:0, pending_elr_cnt:0, contain_hotspot_row:false}) [2022-08-08 11:08:16.902412] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:17.126009] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:17.129185] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:17.129380] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=142] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:17.271084] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] clock out of order(cur_ts=1659928097271078, cur_ts_=1659928097241921, delta=29157) [2022-08-08 11:08:17.616675] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:17.616697] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:17.618546] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:17.618579] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667240, retry interval=1000000, ret=-4179) [2022-08-08 11:08:17.629155] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:17.629220] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667235, retry interval=1000000, ret=-4179) [2022-08-08 11:08:18.131818] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:18.134893] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:18.134947] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=42] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:18.169856] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36D0-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.169970] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36D0-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.170175] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36D0-0-0] [lt=36] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.170346] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36D0-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.170532] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.170730] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.170947] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.171070] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:18.361091] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:18.524312] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25822) [2022-08-08 11:08:18.533431] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=62] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=35742, time_dist: do_cluster_heartbeat=11, generate_cluster_version=35726) [2022-08-08 11:08:18.575912] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=50763) [2022-08-08 11:08:18.576115] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=64013, time_dist: do_cluster_heartbeat=10, generate_cluster_version=64001) [2022-08-08 11:08:18.644584] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=32] [dc=0] clock out of order(cur_ts=1659928098644577, cur_ts_=1659928098622363, delta=22214) [2022-08-08 11:08:18.723622] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=23584) [2022-08-08 11:08:18.735944] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12258) [2022-08-08 11:08:18.740020] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=41693, time_dist: do_cluster_heartbeat=15, generate_cluster_version=41674) [2022-08-08 11:08:18.749204] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=49491, time_dist: generate_cluster_version=49486) [2022-08-08 11:08:18.768146] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:18.768187] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667234, retry interval=1000000, ret=-4179) [2022-08-08 11:08:18.771942] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:18.771979] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667239, retry interval=1000000, ret=-4179) [2022-08-08 11:08:18.824866] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=71] [dc=0] clock out of order(cur_ts=1659928098824855, cur_ts_=1659928098789128, delta=35727) [2022-08-08 11:08:18.944662] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=14242) [2022-08-08 11:08:18.955220] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25752, time_dist: do_cluster_heartbeat=9, generate_cluster_version=25741) [2022-08-08 11:08:18.981942] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=26128) [2022-08-08 11:08:18.995319] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=40302, time_dist: do_cluster_heartbeat=10, generate_cluster_version=40288) [2022-08-08 11:08:19.137359] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:19.157332] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:19.157360] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:19.168368] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=45647) [2022-08-08 11:08:19.170842] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=51007, time_dist: do_cluster_heartbeat=11, generate_cluster_version=50991) [2022-08-08 11:08:19.179172] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10619][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10678) [2022-08-08 11:08:19.181222] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=60496, time_dist: do_cluster_heartbeat=32, generate_cluster_version=60459) [2022-08-08 11:08:19.237361] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659928099237349, cur_ts_=1659928099204064, delta=33285) [2022-08-08 11:08:19.319075] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21949) [2022-08-08 11:08:19.319299] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=257] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23019, time_dist: do_cluster_heartbeat=9, generate_cluster_version=23007) [2022-08-08 11:08:19.353864] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12097) [2022-08-08 11:08:19.354131] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14039, time_dist: do_cluster_heartbeat=9, generate_cluster_version=14027) [2022-08-08 11:08:19.553481] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] clock out of order(cur_ts=1659928099553473, cur_ts_=1659928099530294, delta=23179) [2022-08-08 11:08:19.589832] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24623) [2022-08-08 11:08:19.590179] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=25792, time_dist: do_cluster_heartbeat=4, generate_cluster_version=25785) [2022-08-08 11:08:19.622023] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=12445) [2022-08-08 11:08:19.622392] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=13581, time_dist: generate_cluster_version=13578) [2022-08-08 11:08:19.690702] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928099690694, cur_ts_=1659928099668833, delta=21861) [2022-08-08 11:08:19.716755] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=25950) [2022-08-08 11:08:19.717487] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=27943, time_dist: do_cluster_heartbeat=13, generate_cluster_version=27926) [2022-08-08 11:08:19.741687] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=24892) [2022-08-08 11:08:19.741962] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=26742, time_dist: generate_cluster_version=26740) [2022-08-08 11:08:19.801946] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:19.802243] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=141] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:19.882898] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928099882890, cur_ts_=1659928099860775, delta=22115) [2022-08-08 11:08:19.911583] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:19.943990] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=6740] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:19.944045] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667238, retry interval=1000000, ret=-4179) [2022-08-08 11:08:19.944346] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] task process failed, start retry(max retry time=9223372036854667233, retry interval=1000000, ret=-4179) [2022-08-08 11:08:19.937934] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:08:19.944519] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=6581] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:08:19.946898] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13964) [2022-08-08 11:08:19.983802] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=27768] [dc=0] clock out of order(cur_ts=1659928099983795, cur_ts_=1659928099955314, delta=28481) [2022-08-08 11:08:19.993646] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=38896) [2022-08-08 11:08:19.995794] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=14499, time_dist: do_cluster_heartbeat=6, generate_cluster_version=14488) [2022-08-08 11:08:20.029651] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=18942) [2022-08-08 11:08:20.047922] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=16366) [2022-08-08 11:08:20.049998] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=40078, time_dist: generate_cluster_version=40074) [2022-08-08 11:08:20.130193] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928100130184, cur_ts_=1659928100090111, delta=40073) [2022-08-08 11:08:20.146392] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15082) [2022-08-08 11:08:20.146782] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=16268, time_dist: do_cluster_heartbeat=10, generate_cluster_version=16255) [2022-08-08 11:08:20.147999] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=94] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=15147, time_dist: print_stat=9, do_cluster_heartbeat=2, generate_cluster_version=15131) [2022-08-08 11:08:20.159992] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:20.184686] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:20.184752] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:20.511934] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928100511925, cur_ts_=1659928100487137, delta=24788) [2022-08-08 11:08:20.562316] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=20743) [2022-08-08 11:08:20.562788] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=22913, time_dist: generate_cluster_version=22909) [2022-08-08 11:08:20.565141] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=24150, time_dist: generate_cluster_version=24145) [2022-08-08 11:08:20.936046] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15103520}) [2022-08-08 11:08:20.978262] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=21907) [2022-08-08 11:08:20.992738] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=38] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=44128, time_dist: generate_cluster_version=44125) [2022-08-08 11:08:21.006043] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:21.006104] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667237, retry interval=1000000, ret=-4179) [2022-08-08 11:08:21.014039] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659928101008436 2022-08-08 03:08:21.008436|[init] u=0 arg1:140085233101264, ctx_type:"participant", trans_type:0, trans_id:{hash:1422758383041396313, inc:84126606, addr:{ip:"192.168.11.128", port:2882}, t:1659928101006000}, pkey:{tid:1099511627916, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFC52F81B4-0-0, start_time:1659928101007643, snapshot:1659928100962782|[end_task] u=124 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=10 stc:1659928101008579|[calc_checksum_by_commit] u=20 ret:0, checksum:1701810905, checksum_log_ts:9223372036854775807|[submit_log] u=124 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=5240 trans_version:1659928101006051, uref:1073741825|[exiting] u=16 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:989053, logging:0, pending:0, uref:1|[end_trans_cb] u=5 ret:0, param:0, param:true|[destroy] u=5 arg1:140085233102640|total_timeu=5562) [2022-08-08 11:08:21.036897] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928101036888, cur_ts_=1659928101014330, delta=22558) [2022-08-08 11:08:21.062907] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:21.062945] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667232, retry interval=1000000, ret=-4179) [2022-08-08 11:08:21.189931] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:21.206179] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=66] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:21.206238] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:21.279290] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928098414624, end_time=1659928101279272, elapsed_time=2864648, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:08:21.280655] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36D9-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:21.820812] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10784][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:21.868887] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10787][0][Y0-0000000000000000-0-0] [lt=51] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:21.944382] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18553, time_dist: do_cluster_heartbeat=10, generate_cluster_version=18539) [2022-08-08 11:08:22.037843] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:22.038026] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:22.039750] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=52] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:22.039853] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] task process failed, start retry(max retry time=9223372036854667236, retry interval=1000000, ret=-4179) [2022-08-08 11:08:22.052396] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10748][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:22.066241] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CB1-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:22.066287] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CB1-0-0] [lt=28] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:22.088275] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:22.088300] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:22.093986] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=91] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:22.094201] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=60] [dc=0] task process failed, start retry(max retry time=9223372036854667231, retry interval=1000000, ret=-4179) [2022-08-08 11:08:22.101506] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10723][0][Y0-0000000000000000-0-0] [lt=53] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:22.193384] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10688][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.209502] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:22.213151] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:22.213219] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=47] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:22.267839] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10690][0][Y0-0000000000000000-0-0] [lt=185] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.273106] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10794][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:22.273220] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10752][0][Y0-0000000000000000-0-0] [lt=44] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:22.274171] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10746][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:22.288713] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DC-0-0] [lt=32] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:22.288763] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DC-0-0] [lt=16] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:22.288999] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DC-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:22.289020] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DC-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:22.289139] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:22.342808] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10767][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:22.356845] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10683][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.401417] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10722][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:22.408635] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10676][0][Y0-0000000000000000-0-0] [lt=137] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.418905] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10689][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.550890] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10793][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:22.561945] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10681][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.649627] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10747][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:22.695279] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10691][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:22.719341] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10733][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:22.721658] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=257] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=12561, time_dist: do_cluster_heartbeat=9, generate_cluster_version=12548) [2022-08-08 11:08:22.773189] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10590][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:23.092181] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=4113] [dc=0] clock out of order(cur_ts=1659928103092173, cur_ts_=1659928103065733, delta=26440) [2022-08-08 11:08:23.106929] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:23.106956] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:23.109172] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:23.109230] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667235, retry interval=1000000, ret=-4179) [2022-08-08 11:08:23.120219] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:23.120627] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=254] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:23.124788] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:23.124857] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667230, retry interval=1000000, ret=-4179) [2022-08-08 11:08:23.219594] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:23.224287] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=73] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:23.224333] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=34] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:23.308868] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DF-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.309163] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DF-0-0] [lt=17] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.309446] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DF-0-0] [lt=11] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.309876] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36DF-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.309993] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.310010] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.310183] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:23.310406] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.126797] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CB4-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:24.126940] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CB4-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:24.129819] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=10117) [2022-08-08 11:08:24.130517] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=11565, time_dist: do_cluster_heartbeat=10, generate_cluster_version=11553) [2022-08-08 11:08:24.131809] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:24.131946] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667234, retry interval=1000000, ret=-4179) [2022-08-08 11:08:24.147171] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:24.147328] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:24.148612] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:24.148676] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] task process failed, start retry(max retry time=9223372036854667229, retry interval=1000000, ret=-4179) [2022-08-08 11:08:24.227121] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:24.231368] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:24.231405] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:24.317674] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36E2-0-0] [lt=7] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.317715] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36E2-0-0] [lt=25] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.317902] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36E2-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.318050] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36E2-0-0] [lt=20] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.318163] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.318456] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:24.318601] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:25.061493] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928105061486, cur_ts_=1659928105026173, delta=35313) [2022-08-08 11:08:25.135984] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:25.136052] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=48] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:25.157346] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10620][0][Y0-0000000000000000-0-0] [lt=63] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=18322) [2022-08-08 11:08:25.162855] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=25590, time_dist: generate_cluster_version=25585) [2022-08-08 11:08:25.189972] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=8] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=18308, time_dist: generate_cluster_version=18305) [2022-08-08 11:08:25.196820] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:25.196864] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667228, retry interval=1000000, ret=-4179) [2022-08-08 11:08:25.203235] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=36] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:25.203317] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] task process failed, start retry(max retry time=9223372036854667233, retry interval=1000000, ret=-4179) [2022-08-08 11:08:25.234146] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:25.252325] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:25.252378] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:25.497823] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10732][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:25.607683] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10677][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:25.642166] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10775][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:25.675686] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10744][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:25.821017] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11023][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:25.821067] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10828][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:25.829636] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10837][0][Y0-0000000000000000-0-0] [lt=981] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:25.934366] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11022][0][Y0-0000000000000000-0-0] [lt=100] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:25.935084] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10820][0][Y0-0000000000000000-0-0] [lt=7] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:25.935886] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10765][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:25.953997] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10786][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:25.956588] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10735][0][Y0-0000000000000000-0-0] [lt=35] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:25.974825] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10734][0][Y0-0000000000000000-0-0] [lt=237] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:25.997233] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10764][0][Y0-0000000000000000-0-0] [lt=183] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:26.008964] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10702][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:26.023488] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10682][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.028883] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10750][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.028984] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10751][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.031263] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10686][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.041644] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10731][0][Y0-0000000000000000-0-0] [lt=55] [dc=0] Alloc worker less than lack(num=5, need_num=0, succ_num=0) [2022-08-08 11:08:26.049903] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10745][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.059326] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10705][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:26.101436] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11025][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:26.111763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10838][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:26.120520] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10743][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.122334] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10749][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.122843] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10761][0][Y0-0000000000000000-0-0] [lt=45] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:26.123073] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10714][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:26.126606] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10687][0][Y0-0000000000000000-0-0] [lt=30] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.142920] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10713][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:26.143602] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10762][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:26.148143] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10763][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:26.153238] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10865][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.174402] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10785][0][Y0-0000000000000000-0-0] [lt=40] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.195087] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10776][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:26.195508] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10792][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.202881] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10790][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.218274] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10789][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.225358] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=19] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:26.225394] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] task process failed, start retry(max retry time=9223372036854667232, retry interval=1000000, ret=-4179) [2022-08-08 11:08:26.225493] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10760][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:26.227088] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:26.227284] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10490][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:26.228265] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10674][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.229050] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:26.229302] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=169] [dc=0] task process failed, start retry(max retry time=9223372036854667227, retry interval=1000000, ret=-4179) [2022-08-08 11:08:26.249529] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10819][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:26.255452] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:26.259130] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:26.259183] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=41] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:26.261716] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10692][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.331238] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10685][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.332112] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=33] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:26.332413] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:26.332493] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:26.339657] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10829][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] Alloc worker less than lack(num=2, need_num=0, succ_num=0) [2022-08-08 11:08:26.342420] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10766][0][Y0-0000000000000000-0-0] [lt=24] [dc=0] Alloc worker less than lack(num=8, need_num=0, succ_num=0) [2022-08-08 11:08:26.344694] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10700][0][Y0-0000000000000000-0-0] [lt=13] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:26.368353] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10701][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:26.393720] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10703][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:26.456456] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10678][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.476866] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10675][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.530254] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10684][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.599063] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10704][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] Alloc worker less than lack(num=6, need_num=0, succ_num=0) [2022-08-08 11:08:26.600946] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] clock out of order(cur_ts=1659928106600940, cur_ts_=1659928106568459, delta=32481) [2022-08-08 11:08:26.633487] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10673][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.685062] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10791][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] Alloc worker less than lack(num=10, need_num=0, succ_num=0) [2022-08-08 11:08:26.700054] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10679][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:26.925763] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [10680][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] Alloc worker less than lack(num=20, need_num=0, succ_num=0) [2022-08-08 11:08:27.140408] TRACE [TRACE]ob_trans_ctx.cpp:489 [10630][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] [trans sampling] (TRACE=begin_ts=1659928107139553 2022-08-08 03:08:27.139553|[init] u=0 arg1:140085233546576, ctx_type:"participant", trans_type:0, trans_id:{hash:8433828187136896926, inc:84126833, addr:{ip:"192.168.11.128", port:2882}, t:1659928107138000}, pkey:{tid:1099511628002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=6 ret:0, left_time:10000000, uref:1073741825|[start_task] u=6 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB8ACB4D-0-0, start_time:1659928107139446, snapshot:1659928107110131|[end_task] u=121 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=7 stc:1659928107139691|[calc_checksum_by_commit] u=18 ret:0, checksum:3239696508, checksum_log_ts:9223372036854775807|[submit_log] u=81 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=2 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=538 trans_version:1659928107138287, uref:1073741825|[exiting] u=10 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8318942, logging:0, pending:0, uref:1|[end_trans_cb] u=21 ret:0, param:0, param:true|[destroy] u=3 arg1:140085233547952|total_timeu=816) [2022-08-08 11:08:27.250205] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=22] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:27.250243] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667226, retry interval=1000000, ret=-4179) [2022-08-08 11:08:27.261870] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:27.268190] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:27.268306] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] task process failed, start retry(max retry time=9223372036854667231, retry interval=1000000, ret=-4179) [2022-08-08 11:08:27.270966] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:27.271000] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=23] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:28.216952] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] clock out of order(cur_ts=1659928108216945, cur_ts_=1659928108196439, delta=20506) [2022-08-08 11:08:28.252882] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10491][0][YB42C0A80B80-0005E3CFC62C4CBA-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:28.274210] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:28.283882] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=31] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:28.283946] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] task process failed, start retry(max retry time=9223372036854667225, retry interval=1000000, ret=-4179) [2022-08-08 11:08:28.290243] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:28.290268] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:28.305216] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:28.305253] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667230, retry interval=1000000, ret=-4179) [2022-08-08 11:08:28.342509] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36EE-0-0] [lt=4] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:28.342667] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36EE-0-0] [lt=23] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:28.342713] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][YB42C0A80B80-0005E3CFCABE36EE-0-0] [lt=19] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:28.343959] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10609][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Resource temporarily unavailable") [2022-08-08 11:08:28.385725] WARN [SERVER.OMT] acquire_more_worker (ob_tenant.cpp:1350) [11024][0][Y0-0000000000000000-0-0] [lt=28] [dc=0] Alloc worker less than lack(num=4, need_num=0, succ_num=0) [2022-08-08 11:08:28.590465] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=201] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=15794) [2022-08-08 11:08:28.591390] WARN [LIB] ~ObTimeGuard (utility.h:788) [10841][0][Y0-0000000000000000-0-0] [lt=14] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1, thread_timer_task' cost too much time, used=21599, time_dist: generate_cluster_version=21595) [2022-08-08 11:08:28.624372] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [11016][0][Y0-0000000000000000-0-0] [lt=16] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_batch_buffer.cpp", line=378, time=14805) [2022-08-08 11:08:28.630486] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=18] [dc=0] clock out of order(cur_ts=1659928108630477, cur_ts_=1659928108609032, delta=21445) [2022-08-08 11:08:28.643763] WARN [CLOG] finish_timer (ob_log_timer_utility.h:37) [10621][0][Y0-0000000000000000-0-0] [lt=188] [dc=0] cost too much time(file="/home/admin/181_20220110212854347_249556029_code/rpm_workspace/src/clog/ob_clog_file_writer.cpp", line=670, time=13215) [2022-08-08 11:08:28.643971] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=17] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=47653, time_dist: generate_cluster_version=47650) [2022-08-08 11:08:28.715863] WARN [LIB] ~ObTimeGuard (utility.h:788) [11016][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=14955, time_dist: do_cluster_heartbeat=10, generate_cluster_version=14943) [2022-08-08 11:08:29.291497] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:29.291545] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10492][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:29.293450] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:29.310468] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=26] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:29.316457] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=5972] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:29.323866] WARN stop_server (ob_root_service.cpp:9277) [10492][0][Y0-0000000000000000-0-0] [lt=9] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:29.323915] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10492][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667224, retry interval=1000000, ret=-4179) [2022-08-08 11:08:29.326040] WARN [SHARE] io_cancel (ob_local_device.cpp:1126) [10493][0][Y0-0000000000000000-0-0] [lt=5] [dc=0] Fail to cancel aio, (ret=-4009, sys_ret=-22, errmsg="Connection timed out") [2022-08-08 11:08:29.328699] WARN stop_server (ob_root_service.cpp:9277) [10493][0][Y0-0000000000000000-0-0] [lt=10] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:29.328730] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10493][0][Y0-0000000000000000-0-0] [lt=2] [dc=0] task process failed, start retry(max retry time=9223372036854667229, retry interval=1000000, ret=-4179) [2022-08-08 11:08:29.763789] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=15] [dc=0] clock out of order(cur_ts=1659928109763779, cur_ts_=1659928109736119, delta=27660) [2022-08-08 11:08:29.853109] WARN [STORAGE] set_io_prohibited (ob_storage.cpp:111) [10585][0][Y0-0000000000000000-0-0] [lt=6] [dc=0] set_io_prohibited(io_prohibited=false, prohibited=false) [2022-08-08 11:08:29.853150] WARN [STORAGE] enable_backup_white_list (ob_partition_service.cpp:14353) [10585][0][Y0-0000000000000000-0-0] [lt=39] [dc=0] backup set_io_prohibited(prohibited=false) [2022-08-08 11:08:30.028626] WARN [STORAGE.TRANS] run1 (ob_clock_generator.cpp:85) [10569][0][Y0-0000000000000000-0-0] [lt=20] [dc=0] clock out of order(cur_ts=1659928110028619, cur_ts_=1659928110004703, delta=23916) [2022-08-08 11:08:30.118685] WARN [CLOG.CSR] operator() (ob_ilog_store.cpp:121) [10622][0][Y0-0000000000000000-0-0] [lt=29] [dc=0] invalid partition(partition_key={tid:1100611139453777, partition_id:0, part_cnt:0}, index_info_block_entry={min_log_id:250, max_log_id:259, min_log_timestamp:1659895058001672, max_log_timestamp:1659926821403720, start_offset_:15107936}) [2022-08-08 11:08:30.172967] TRACE [TRACE]ob_trans_ctx.cpp:489 [10629][0][YB42C0A80B80-0005E3CF97E8361E-0-0] [lt=24] [dc=0] [trans sampling] (TRACE=begin_ts=1659928110171459 2022-08-08 03:08:30.171459|[init] u=0 arg1:140085232374224, ctx_type:"participant", trans_type:0, trans_id:{hash:12054214084990834608, inc:84126946, addr:{ip:"192.168.11.128", port:2882}, t:1659928110170000}, pkey:{tid:1100611139404002, partition_id:0, part_cnt:0}, uref:1073741824|[start_trans] u=7 ret:0, left_time:10000000, uref:1073741825|[start_task] u=5 ret:0, sql_no:4294967297, uref:1073741825, trace_id:YB42C0A80B80-0005E3CFCB9A6D8A-0-0, start_time:1659928110171307, snapshot:1659928110130289|[end_task] u=126 ret:0, is_rollback:false, uref:1073741825|[set_stc] u=8 stc:1659928110171602|[calc_checksum_by_commit] u=21 ret:0, checksum:293695560, checksum_log_ts:9223372036854775807|[submit_log] u=27 ret:0, log_type:64, logging:1, pending:0, uref:1073741825|[sp_commit] u=1 ret:0, is_rollback:false, is_readonly:false, uref:1073741825|[update_trans_version] u=1220 trans_version:1659928110171624, uref:1073741825|[exiting] u=26 uref:1073741825, arg1:1, arg2:1|[on_sync_log_succ] u=3 ret:0, log_type:64, log_id:8304360, logging:0, pending:0, uref:1|[end_trans_cb] u=17 ret:0, param:0, param:true|[destroy] u=4 arg1:140085232375600|total_timeu=1465) [2022-08-08 11:08:30.318544] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:118) [10644][0][Y0-0000000000000000-0-0] [lt=11] [dc=0] max file does not exist(max_file_id=22, b_exist=false) [2022-08-08 11:08:30.326449] WARN run1 (ob_timer.cpp:357) [10622][0][Y0-0000000000000000-0-0] [lt=46] [dc=0] timer task cost too much time(task="tasktype:N9oceanbase4clog11ObIlogStore14IlogWriterTaskE", start_time=1659928108416699, end_time=1659928110326437, elapsed_time=1909738, this=0x7f69b154b010, thread_id=10622) [2022-08-08 11:08:30.328728] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:881) [10644][0][Y0-0000000000000000-0-0] [lt=27] [dc=0] get pg key error(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, pg_key={tid:18446744073709551615, partition_id:-1, part_idx:268435455, subpart_idx:268435455}) [2022-08-08 11:08:30.328969] WARN [STORAGE] get_partition (ob_partition_meta_redo_module.cpp:858) [10644][0][Y0-0000000000000000-0-0] [lt=226] [dc=0] fail to get partition(ret=-4225, pkey={tid:1100611139453777, partition_id:0, part_cnt:0}, common::lbt()="0xc021167 0x78e5eb4 0x3b4142c 0x3d4b00d 0x6beb935 0x6c2eb22 0x6c2f238 0x6b8e2ea 0x6b8af44 0x3cd08cb 0x4350998 0xbe5dda4 0xbe5d952 0xbe54b2f") [2022-08-08 11:08:30.363989] WARN stop_server (ob_root_service.cpp:9277) [10491][0][Y0-0000000000000000-0-0] [lt=25] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:30.364029] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10491][0][Y0-0000000000000000-0-0] [lt=3] [dc=0] task process failed, start retry(max retry time=9223372036854667223, retry interval=1000000, ret=-4179) [2022-08-08 11:08:30.375073] WARN stop_server (ob_root_service.cpp:9277) [10490][0][Y0-0000000000000000-0-0] [lt=12] [dc=0] not enough member or quorum mismatch, stop servers not allowed [2022-08-08 11:08:30.375107] WARN [SHARE] run2 (ob_async_task_queue.cpp:144) [10490][0][Y0-0000000000000000-0-0] [lt=4] [dc=0] task process failed, start retry(max retry time=9223372036854667228, retry interval=1000000, ret=-4179)